Catalog
method#Data#QA#Integration#Observability

Data Testing

A methodical approach to systematically test data quality, transformations and data pipelines using automated tests and validations.

Data testing is a methodical approach to systematically verify data quality and data pipelines using automated tests, validations, and contract checks.
Emerging
Medium

Classification

  • Medium
  • Technical
  • Architectural
  • Intermediate

Technical context

Great Expectationsdbt (Data Build Tool)Apache Airflow / other orchestrators

Principles & goals

Shift-left: embed tests as early as possibleAutomation: run tests in CI/CDDefine contracts: clear schema and API contracts
Build
Team, Domain

Use cases & scenarios

Compromises

  • False test coverage leads to false sense of security
  • Performance tests on production data can have side effects
  • Excessive alerts lead to alert fatigue
  • Test data anonymization and versioning
  • Prioritize critical paths and metrics
  • Provide failures with reproducible examples

I/O & resources

  • Schema/contract specifications
  • Representative test or production data (anonymized)
  • Pipeline definitions and transformation logic
  • Detailed test reports and dashboards
  • Failing examples and reproduction datasets
  • Data quality metrics and trend analyses

Description

Data testing is a methodical approach to systematically verify data quality and data pipelines using automated tests, validations, and contract checks. It detects inconsistencies, regressions, and integration errors early in the development cycle. The method covers test design, execution, monitoring, and governance to ensure reliable data products.

✔

  • Early error detection and reduction of regressions
  • Higher reliability of reporting and models
  • Improved collaboration between producers and consumers

✖

  • Requires representative test data and maintenance effort
  • Not all data issues are deterministically testable
  • Initial implementation effort can be high

  • Test coverage

    Share of tested metrics/transformations relative to the total scope.

  • Defect density

    Number of detected data defects per data volume or pipeline run.

  • Mean Time to Detect (MTTD)

    Average time from defect occurrence to detection.

Case study: ETL pipeline tests in retail

A retail team introduced data testing to secure price calculations and aggregations during deployments.

Case study: contract testing between teams

Two teams established contract checks to prevent breaking changes in shared data flows.

Proof of concept: monitoring critical KPIs

PoC implemented automated quality rules and significantly reduced data-related incidents.

1

Identify stakeholders and define quality goals

2

Prioritize critical metrics and test cases

3

Introduce test infrastructure and tools (e.g., Great Expectations)

4

Integrate tests into CI/CD and run on PRs

5

Set up monitoring and alerts for production data

6

Establish regular review and maintenance processes

⚠️ Technical debt & bottlenecks

  • Non-versioned test suites and inconsistent rules
  • Monolithic test pipelines without modularization
  • Missing mock or subsetting strategies for large datasets
Schema evolutionData volumeETL complexity
  • Tests that run only on small synthetic datasets
  • Blocking deployments entirely due to low-severity quality warnings
  • Using production data without anonymization
  • Lack of test data maintenance leads to false negatives
  • Excessive test data sizes significantly slow down CI
  • Unclear responsibilities for data tests between teams
SQL and data modeling skillsExperience with data engineering and ETL processesKnowledge in test design and automation
Data qualityPipeline reliabilityObservability and monitoring
  • • Availability of representative test data
  • • Data privacy and compliance requirements
  • • Limited tooling support in legacy environments