Data quality as code: contracts, expectations, and stopping bad data at the door

A data pipeline can run successfully, pass every scheduled job, update every dashboard, and still produce completely incorrect business decisions.

Imagine an e-commerce platform that receives order data from an upstream service. For months, a field called gross_total_cents represents the total value of purchased items before discounts.

One morning, the upstream team changes the calculation. The field now contains the discounted amount, but its name and data type remain unchanged.

Every SQL query still executes. Every scheduled job finishes. Every dashboard refreshes.

Yet the company’s revenue reports are now measuring something different.

This is one of the most dangerous forms of data quality failure: the data remains technically valid while its business meaning changes.

A conventional schema validator might never notice.

The solution is to treat data quality as code: define explicit contracts, translate those contracts into executable expectations, test how failures behave, and prevent untrusted data from reaching downstream consumers.

In this hands-on guide, we’ll build a version-controlled validation system using Python, Pandera, pandas, PyYAML, and pytest. We’ll implement strict CSV ingestion, detect structural and semantic drift, prevent invalid batches from replacing trusted data, and integrate the checks into continuous integration.

The tutorial uses Pandera 0.34.1, released October 6, 2026, and pytest 9.1.1, released June 19, 2026. We’ll also use the pandas 3.0 series and current GitHub Actions v7 releases. These versions reflect the available stable tooling as of October 2026.

By the end, you’ll understand not only how to write validation rules, but how to make those rules an enforceable boundary between untrusted source systems and trusted analytical data.

1. Why successful pipelines can still produce bad data

Traditional pipeline monitoring concentrates on operational success.

A typical workflow looks like this:

Upstream service
    |
    v
CSV / API / Event stream
    |
    v
Ingestion
    |
    v
Staging and transformations
    |
    v
Data warehouse
    |
    v
Dashboards / Reports / ML

Operational monitoring usually answers questions such as:

• Did the ingestion process finish?

• Did the database query execute?

• Did the transformation job succeed?

• Was the output table updated?

• Did the scheduler report an error?

These are necessary checks, but they establish that software executed—not that the resulting information is correct.

Data quality engineering asks a different set of questions:

• Are all required fields present?

• Do values conform to the agreed types and formats?

• Are identifiers unique at the correct business grain?

• Are numerical relationships internally consistent?

• Do values still represent the same business concepts?

• Is the dataset complete and sufficiently fresh?

• Should this incoming batch be allowed into production?

The most useful distinction is between three forms of drift.

Structural drift is often easy to identify because something visible changes in the schema.

Representation drift can be more subtle. An ingestion tool might automatically convert a text value into a number and conceal a producer-side format change.

Semantic drift is especially difficult because an incorrect value can remain perfectly compatible with the schema.

The key principle is that a schema describes what data looks like; a contract must also describe what data means.

2. Data contracts and expectations solve different problems

Data contracts and data expectations are closely related, but they are not interchangeable.

A data contract defines the agreement

A data contract records what an upstream producer promises and what its consumers may safely assume.

A useful contract describes:

• Dataset name and contract version.

• Producer and consumer ownership.

• Row-level grain and identifier uniqueness.

• Field names, types, and nullability.

• Accepted values and numerical bounds.

• Units, currencies, and time-zone conventions.

• Business definitions and calculation rules.

• Freshness and completeness requirements.

• Breaking-change and compatibility policies.

• Failure handling and escalation ownership.

For example, a financial field should not merely be described as an integer.

Its definition should clarify whether it represents gross sales, net sales, recognized revenue, refunded amounts, or something else entirely.

Likewise, a date field needs a business interpretation: UTC calendar date, customer-local date, or reporting-period date.

These details are part of the interface between teams.

Data expectations make the agreement executable

An expectation is a statement that software can verify against an actual dataset.

For a retail order feed, useful expectations include:

• Every order line has a nonempty identifier.

• Each order-line identifier appears at most once per batch.

• Quantities are positive integers.

• Currency is USD.

• The declared contract version is supported.

• Gross totals equal quantity multiplied by unit price.

• The incoming batch contains at least one record.

The relationship can be expressed plainly:

gross_total_cents = quantity × unit_price_cents

If the source starts sending discounted totals in the gross-total field, this equation may fail even though all the structural checks continue to pass.

A contract specifies the agreement. Expectations enforce measurable parts of it. Regression tests verify that the enforcement behaves correctly when the contract is violated.

What validation cannot prove

No validation library can independently establish every business meaning.

Suppose an upstream system changes quantity, unit price, and gross total together while preserving the arithmetic relationship. The dataset might remain internally consistent even though its interpretation has changed.

Detecting that situation may require independent source reconciliation, explicit producer versioning, historical comparisons, or governance controls.

This limitation matters: executable expectations reduce risk, but they do not eliminate the need for accountable producer-consumer agreements.

3. Choosing a modern data validation stack

Several complementary tools can support data quality engineering.

We’ll use Pandera as the validation engine and pytest as the regression-testing framework.

Pandera’s supported pandas-specific import is:

import pandera.pandas as pa

The project has supported pandas 3 since its 0.30.0 release, and its current API provides dataframe schemas, column checks, dataframe-wide expectations, strict column enforcement, and lazy error reporting.

We’ll also address a common maintenance problem: storing the contract in YAML while independently hard-coding another version of the rules in Python.

Instead, the executable schema will be constructed from the YAML contract. This makes the document the authoritative source for field names, constraints, and accepted values, while Python implements the supported validation mechanisms.

4. Model the dataset at the correct grain

Before writing validation code, we need to establish what one row represents.

Our dataset contains retail order lines, not complete orders. One customer order may contain several different products, and each product line has its own quantity and unit price.

That distinction affects uniqueness.

Two rows may share the same order_id. However, their order_line_id values must differ.

For this tutorial:

• All monetary values use US cents.

• All transactions use USD.

• Prices and quantities are nonnegative where applicable.

• Quantity must be at least one.

• Gross totals exclude discounts, taxes, and shipping.

• Each row represents one order line.

• The feed contains complete snapshots, not incremental updates.

Our principal invariant is:

gross_total_cents = quantity × unit_price_cents

For example, an order line containing two products at 1,200 cents each must have a gross total of 2,400 cents.

A real order-management platform may have additional complexities such as bundles, quantity-based discounts, partial shipments, or price adjustments. Those would require a more detailed contract.

The important rule is to validate against the producer’s actual business semantics, not an oversimplified assumption.

5. Define the contract in YAML

We’ll use a custom YAML contract that is both readable by engineers and interpreted by the validation application.

Create orders_contract.yaml.

dataset: retail_order_lines
version: "1.0.0"
grain: one_order_line

owners:
  producer: commerce-platform
  consumer: analytics-platform

fields:
  order_id:
    type: string
    nullable: false
    pattern: '^ord-[0-9]{4,8}$'
    meaning: identifier_of_parent_order

  order_line_id:
    type: string
    nullable: false
    unique: true
    pattern: '^line-[0-9]{4,8}$'
    meaning: identifier_of_order_line

  contract_version:
    type: string
    nullable: false
    allowed: [v1]

  quantity:
    type: integer
    nullable: false
    wire_format: canonical_unsigned_decimal
    min: 1
    max: 1000

  unit_price_cents:
    type: integer
    nullable: false
    wire_format: canonical_unsigned_decimal
    min: 0
    max: 5000000
    unit: USD_cents

  gross_total_cents:
    type: integer
    nullable: false
    wire_format: canonical_unsigned_decimal
    min: 0
    max: 5000000000
    unit: USD_cents
    meaning: pre_discount_pre_tax_line_extension

  currency:
    type: string
    nullable: false
    allowed: [USD]

  status:
    type: string
    nullable: false
    allowed: [paid, shipped, refunded]

invariant:
  kind: gross_line_extension
  output: gross_total_cents
  quantity: quantity
  unit_price: unit_price_cents

policy:
  on_validation_failure: reject_batch
  on_semantic_change: require_review_and_new_version

This file captures structural constraints, business definitions, and operational policy.

The wire_format setting is particularly important.

CSV files contain textual representations of values. A token such as 2400 might reasonably represent an integer monetary amount, while 2400.0 is a decimal representation.

Even if both could be converted to the same numerical value, we may want to reject the unexpected representation because it indicates an upstream format change.

Our contract therefore requires canonical unsigned decimal integers: no currency symbols, commas, decimal points, leading zeros, signs, or surrounding whitespace.

Preventing contract documentation drift

A common implementation mistake is to define constraints in YAML and then duplicate them manually in Python.

Suppose the YAML declares a maximum quantity of 1,000 while the Python schema allows 10,000. Which definition is correct?

We’ll avoid this disagreement by generating Pandera constraints from the YAML values.

Our application will interpret the supported field rules, reject unknown validation-rule names, and construct the executable schema. The supported cross-column invariant will also be configured by the YAML.

This does not make arbitrary YAML an automatically supported validation language. Our implementation deliberately recognizes a small vocabulary of rules.

For organizations managing many contract types, a standardized specification such as the Open Data Contract Standard may be appropriate.

For this tutorial, the smaller custom format keeps the relationship between documentation and enforcement visible.

6. Create a reproducible project structure

Use Python 3.13 for the walkthrough.

Create the project and its virtual environment.

mkdir data-quality-contracts
cd data-quality-contracts

python -m venv .venv
source .venv/bin/activate

python -m pip install --upgrade pip

On Windows PowerShell, activate the environment with the appropriate activation script for that shell.

Create a dependency manifest named requirements.txt.

pandera[pandas]==0.34.1
pandas==3.0.6
pytest==9.1.1
PyYAML==6.0.3

These are exact top-level dependency pins rather than open-ended minimum-version constraints.

Pandera 0.34.1 is the stable release published October 6, 2026. pandas 3.0.6 was published September 17, 2026, and pytest 9.1.1 was published June 19, 2026.

Install the dependencies.

python -m pip install -r requirements.txt

Check the installed versions.

python -m pip show pandera pandas pytest PyYAML

The intended project layout is:

data-quality-contracts/
├── .github/
│   └── workflows/
│       └── data-quality.yml
├── incoming/
│   └── orders.csv
├── trusted/
├── orders_contract.yaml
├── requirements.txt
├── quality.py
└── test_quality.py

Each component has a clear responsibility.

• incoming/ contains untrusted producer data.

• trusted/ contains successfully validated output.

• orders_contract.yaml defines the approved agreement.

• quality.py implements contract interpretation, validation, and publication.

• test_quality.py exercises successful and unsuccessful behavior.

• requirements.txt supplies consistent direct dependency versions.

• The GitHub Actions workflow runs the tests in CI.

The dependency manifest makes installation repeatable at the direct-dependency level. For stronger reproducibility, generate and commit a fully resolved lockfile with transitive dependency versions and, where appropriate, artifact hashes.

7. Build the executable validator

Create quality.py.

The implementation has five responsibilities:

1. Load and check the contract definition.

2. Detect structural differences before evaluating business invariants.

3. Construct a Pandera schema from the contract.

4. Parse incoming CSV values without silently normalizing malformed monetary amounts.

5. Publish only successfully validated data.

import argparse
import csv
import os
import re
import sys
import tempfile
from pathlib import Path

import pandas as pd
import pandera.pandas as pa
import yaml


CONTRACT_PATH = Path(__file__).with_name(
    "orders_contract.yaml"
)

INTEGER_TOKEN = re.compile(
    r"(?:0|[1-9][0-9]*)\Z",
    flags=re.ASCII,
)

FIELD_OPTIONS = {
    "type",
    "nullable",
    "unique",
    "pattern",
    "allowed",
    "min",
    "max",
    "unit",
    "meaning",
    "wire_format",
}


def load_contract(path=CONTRACT_PATH):
    contract = yaml.safe_load(
        Path(path).read_text(encoding="utf-8")
    )

    if (
        not isinstance(contract, dict)
        or contract.get("grain") != "one_order_line"
    ):
        raise ValueError(
            "Unsupported or invalid data contract"
        )

    fields = contract.get("fields")

    if not isinstance(fields, dict) or not fields:
        raise ValueError(
            "Contract must declare fields"
        )

    for name, rule in fields.items():
        if not isinstance(rule, dict):
            raise ValueError(
                f"Invalid field rule: {name}"
            )

        unknown = set(rule) - FIELD_OPTIONS

        if unknown:
            raise ValueError(
                f"Unsupported rules for {name}: "
                f"{sorted(unknown)}"
            )

        if rule.get("type") not in {
            "string",
            "integer",
        }:
            raise ValueError(
                f"Unsupported type for {name}"
            )

        if (
            rule["type"] == "integer"
            and rule.get("wire_format")
            != "canonical_unsigned_decimal"
        ):
            raise ValueError(
                f"Missing integer wire format for {name}"
            )

    invariant = contract.get("invariant")

    if (
        not isinstance(invariant, dict)
        or invariant.get("kind")
        != "gross_line_extension"
    ):
        raise ValueError(
            "Unsupported invariant"
        )

    if any(
        invariant.get(key) not in fields
        for key in (
            "output",
            "quantity",
            "unit_price",
        )
    ):
        raise ValueError(
            "Invariant references an unknown field"
        )

    return contract


def require_columns(actual, contract):
    actual = list(actual)
    expected = list(contract["fields"])

    if len(actual) != len(set(actual)):
        raise ValueError(
            "Duplicate column names"
        )

    missing = sorted(
        set(expected) - set(actual)
    )

    extra = sorted(
        set(actual) - set(expected)
    )

    if missing or extra:
        raise ValueError(
            f"Schema drift: missing={missing}; "
            f"unexpected={extra}"
        )


def build_schema(contract):
    columns = {}

    for name, rule in contract["fields"].items():
        checks = []

        if "min" in rule:
            checks.append(
                pa.Check.ge(rule["min"])
            )

        if "max" in rule:
            checks.append(
                pa.Check.le(rule["max"])
            )

        if "allowed" in rule:
            checks.append(
                pa.Check.isin(rule["allowed"])
            )

        if "pattern" in rule:
            checks.append(
                pa.Check.str_matches(
                    rule["pattern"]
                )
            )

        columns[name] = pa.Column(
            int if rule["type"] == "integer" else str,
            checks=checks,
            nullable=rule.get(
                "nullable", False
            ),
            unique=rule.get(
                "unique", False
            ),
        )

    invariant = contract["invariant"]

    check = pa.Check(
        lambda df: (
            df[invariant["output"]]
            == (
                df[invariant["quantity"]]
                * df[invariant["unit_price"]]
            )
        ),
        error=(
            "Gross total differs from "
            "quantity times unit price"
        ),
    )

    return pa.DataFrameSchema(
        columns=columns,
        checks=[check],
        strict=True,
        coerce=False,
        unique_column_names=True,
    )


def validate_orders(
    df,
    contract_path=CONTRACT_PATH,
):
    contract = load_contract(contract_path)

    require_columns(
        df.columns,
        contract,
    )

    if df.empty:
        raise ValueError(
            "Order batch must not be empty"
        )

    return build_schema(
        contract
    ).validate(
        df,
        lazy=True,
    )


def parse_integer(
    token,
    column,
    line_number,
):
    if (
        not isinstance(token, str)
        or not INTEGER_TOKEN.fullmatch(token)
    ):
        raise ValueError(
            f"Line {line_number}: "
            f"invalid integer token in "
            f"{column}: {token!r}"
        )

    if (
        len(token) > 19
        or int(token) > 9_223_372_036_854_775_807
    ):
        raise ValueError(
            f"Line {line_number}: "
            f"integer overflow in {column}"
        )

    return int(token)


def read_order_csv(path, contract):
    with Path(path).open(
        "r",
        encoding="utf-8-sig",
        newline="",
    ) as handle:
        reader = csv.reader(
            handle,
            strict=True,
        )

        header = next(reader, None)

        if header is None:
            raise ValueError(
                "CSV has no header"
            )

        require_columns(
            header,
            contract,
        )

        rows = []

        for line_number, row in enumerate(
            reader,
            start=2,
        ):
            if len(row) != len(header):
                raise ValueError(
                    f"Line {line_number}: "
                    "wrong field count"
                )

            rows.append(row)

    df = pd.DataFrame(
        rows,
        columns=header,
    )

    for name, rule in contract["fields"].items():
        if rule["type"] == "integer":
            df[name] = [
                parse_integer(
                    token,
                    name,
                    line_number,
                )
                for line_number, token in enumerate(
                    df[name],
                    start=2,
                )
            ]

    return df


def publish_orders(
    source,
    destination,
    contract_path=CONTRACT_PATH,
):
    contract = load_contract(
        contract_path
    )

    df = read_order_csv(
        source,
        contract,
    )

    validated = validate_orders(
        df,
        contract_path,
    )

    destination = Path(destination)

    destination.parent.mkdir(
        parents=True,
        exist_ok=True,
    )

    temporary = None

    try:
        with tempfile.NamedTemporaryFile(
            mode="w",
            encoding="utf-8",
            newline="",
            dir=destination.parent,
            prefix=".orders-",
            suffix=".tmp",
            delete=False,
        ) as handle:
            temporary = Path(handle.name)

            validated.to_csv(
                handle,
                index=False,
            )

            handle.flush()
            os.fsync(handle.fileno())

        os.replace(
            temporary,
            destination,
        )

    finally:
        if temporary is not None:
            temporary.unlink(
                missing_ok=True
            )

    print(
        f"PUBLISHED {len(validated)} rows "
        f"to {destination}"
    )


def main():
    parser = argparse.ArgumentParser(
        description=(
            "Validate and publish order lines"
        )
    )

    parser.add_argument(
        "input",
        type=Path,
    )

    parser.add_argument(
        "output",
        type=Path,
    )

    args = parser.parse_args()

    try:
        publish_orders(
            args.input,
            args.output,
        )

    except (
        ValueError,
        csv.Error,
        OSError,
        yaml.YAMLError,
        pa.errors.SchemaError,
        pa.errors.SchemaErrors,
    ) as error:
        print(
            f"REJECTED: {error}",
            file=sys.stderr,
        )
        return 1

    return 0


if __name__ == "__main__":
    sys.exit(main())

This implementation is larger than a minimal dataframe schema, but the additional code protects boundaries that ordinary dataframe validation cannot fully cover.

Let’s examine why those choices matter.

Strict schema enforcement

Pandera is configured with strict schema checking.

Unexpected columns are rejected rather than silently carried into the trusted dataset.

We also check column names before invoking Pandera.

That extra preflight check is important because our dataframe-wide invariant references specific columns.

If an upstream source removes gross_total_cents, evaluating the arithmetic relationship immediately could otherwise raise a missing-column exception before the application produces an understandable structural-drift diagnostic.

Our validation order is therefore:

Load approved contract
        |
        v
Check field names and duplicates
        |
        v
Check that batch is nonempty
        |
        v
Validate types and column rules
        |
        v
Validate cross-column invariant
        |
        v
Accept or reject batch

A missing column fails at the structural boundary, before the business-level validator accesses it.

No silent dataframe coercion

The Pandera schema uses coerce=False.

That prevents Pandera from automatically converting an incompatible dataframe dtype into the expected type.

For example, a dataframe containing floating-point values where integers are required should not be silently accepted through conversion.

However, there is an important limitation to understand.

Disabling Pandera coercion does not disable parsing or type inference performed earlier in the pipeline.

If an application first calls a CSV reader that automatically converts source tokens into numerical values, Pandera receives the already-converted dataframe.

Our implementation addresses this by reading CSV tokens as text using Python’s standard CSV parser, validating monetary representations explicitly, and converting only approved integer tokens.

Why canonical integer parsing matters

Our parser accepts inputs such as:

0
350
1200
2400
5000000

It rejects representations such as:

02400
2,400
2400.0
2.4e3
$24.00
NA

These rules intentionally enforce a source representation rather than merely checking whether Python could extract a number.

CSV does not inherently distinguish typed integer values from typed strings. A quoted CSV field containing 2400 and an unquoted field containing 2400 both decode to the same text token.

Therefore, the contract accepts the canonical textual representation of an integer. It cannot enforce JSON-style distinctions between a numeric scalar and a string scalar using ordinary CSV decoding alone.

If the producer must distinguish those types at the wire level, choose a serialization format with explicit types or inspect the raw format under a separately defined transport contract.

Preventing integer overflow

Our raw parser also rejects values exceeding the signed 64-bit integer range.

This is separate from Pandera’s business-level maximums.

For example, the contract limits unit_price_cents to 5,000,000, which is much smaller than the maximum representable signed 64-bit integer.

The parser enforces representability; Pandera enforces the approved business range.

Keeping these responsibilities separate prevents accidental overflows or implicit conversions before the business rules execute.

Contract-generated rules

The schema builder reads numerical limits, accepted values, string patterns, nullability, and uniqueness directly from YAML.

It also rejects unsupported field-rule keys instead of ignoring them.

This prevents an important category of configuration mistake: adding a new-looking validation rule to the contract that the execution engine never enforces.

Metadata such as ownership and business meaning remains descriptive. The executable checks implement the supported subset of the contract.

Safe local publication

The publication function writes a temporary file in the trusted output directory and then uses os.replace() to publish it.

For a single writer on a suitable local filesystem, this avoids exposing a partially written replacement.

It also ensures that validation failures occur before the trusted destination is touched.

The file is flushed and synchronized before replacement.

This is a useful local demonstration, not a complete distributed transaction protocol. Object stores, distributed filesystems, warehouses, and concurrent writers require mechanisms appropriate to those systems.

8. Run the first successful validation

Create the incoming and trusted directories.

mkdir -p incoming trusted

Create incoming/orders.csv.

cat > incoming/orders.csv <<'CSV'
order_id,order_line_id,contract_version,quantity,unit_price_cents,gross_total_cents,currency,status
ord-0001,line-0001,v1,2,1200,2400,USD,paid
ord-0001,line-0002,v1,1,350,350,USD,shipped
CSV

Notice that both rows share the same parent order ID.

This is valid because the dataset’s grain is one order line, and each line has its own unique identifier.

Execute the validation gate.

python quality.py incoming/orders.csv trusted/orders.csv

For this batch, the expected output is:

PUBLISHED 2 rows to trusted/orders.csv

The first line has:

• Quantity: 2

• Unit price: 1,200 cents

• Gross line total: 2,400 cents

The second line has:

• Quantity: 1

• Unit price: 350 cents

• Gross line total: 350 cents

Both satisfy the invariant.

The combined gross value is 2,750 cents, or USD 27.50.

The trusted output should now contain the validated records.

9. Detect a structural schema drift

Now simulate an upstream producer changing the name of the gross-total column.

The original field is gross_total_cents. The producer replaces it with net_total_cents.

Generate a drifted CSV using a Bash heredoc.

python - <<'PY'
import pandas as pd

df = pd.read_csv("incoming/orders.csv")

drifted = df.rename(
    columns={
        "gross_total_cents": "net_total_cents"
    }
)

drifted.to_csv(
    "incoming/renamed_orders.csv",
    index=False,
)
PY

Run the gate against this modified input.

python quality.py \
    incoming/renamed_orders.csv \
    trusted/orders.csv

The command should exit with a nonzero status.

The relevant diagnostic will resemble:

REJECTED: Schema drift: missing=['gross_total_cents']; unexpected=['net_total_cents']

This is a deliberate failure.

The validator does not assume that the newly named field is equivalent to the missing one.

It does not rename the column automatically or fill the missing gross total with zeros.

Instead, it requires the producer and consumer to agree on the new interface.

This matters because a renamed field could represent either a harmless structural change or a meaningful change in business calculation.

The pipeline cannot determine compatibility from a similar-looking column name.

10. Catch semantic drift without a schema change

Our second experiment demonstrates the more dangerous failure.

The producer continues to send gross_total_cents, but a discount is applied before populating the field.

For the first order line, the correct gross amount is 2,400 cents. Suppose the producer incorrectly supplies 2,000 cents after applying a 400-cent discount.

Generate that dataset.

python - <<'PY'
import pandas as pd

df = pd.read_csv("incoming/orders.csv")

df.loc[
    df["order_line_id"] == "line-0001",
    "gross_total_cents",
] = 2000

df.to_csv(
    "incoming/semantic_drift.csv",
    index=False,
)
PY

Now execute the validation gate.

python quality.py \
    incoming/semantic_drift.csv \
    trusted/orders.csv

This time:

• Every required column exists.

• The identifiers are correctly formatted.

• The monetary amount is an integer.

• The currency is accepted.

• The contract version is recognized.

• The status is valid.

Nevertheless, the gross-total invariant fails.

Running the command prints Pandera’s lazy-validation report (pandera 0.34.1, pandas 3.0.6):

REJECTED: {
    "DATA": {
        "DATAFRAME_CHECK": [
            {
                "schema": null,
                "column": null,
                "check": "Gross total differs from quantity times unit price",
                "error": "DataFrameSchema 'None' failed element-wise validator number 0: <Check <lambda>: Gross total differs from quantity times unit price> failure cases: ord-0001, line-0001, v1, 2, 1200, 2000, USD, paid"
            }
        ]
    }
}

The report names the violated check and the offending row: 2 × 1200 = 2,400 expected, 2,000 received.

The important feature is that the validator rejects a value that satisfies ordinary structural constraints but violates an independent business relationship.

That is precisely why cross-column expectations are valuable.

A limitation worth testing separately

Suppose the upstream producer changes both unit_price_cents and gross_total_cents consistently.

If quantity remains two, unit price becomes 1,000 cents, and gross total becomes 2,000 cents, the arithmetic relationship still holds.

The validator cannot know whether the new price is legitimate or whether the producer has silently redefined unit price after discounts.

To address that situation, production systems may need:

• Independent price-catalog reconciliation.

• Source-side contract versioning.

• Producer change approval.

• Historical distribution comparisons.

• Separate discount and net-total fields.

• Reconciliation against authoritative financial records.

An invariant proves only the relationship it tests. It does not automatically prove that each input reflects the intended external reality.

11. Test the ingestion boundary against malformed values

One frequently overlooked source of bugs is permissive CSV parsing.

For example, a value such as 2400.0 may be converted to a floating-point number by an upstream parser.

Another component might later cast it to an integer.

By the time the data reaches the validation layer, evidence of the original representation has disappeared.

Our design checks the wire representation first.

Generate an input with an unexpected decimal value.

python - <<'PY'
import pandas as pd

df = pd.read_csv("incoming/orders.csv")

df["gross_total_cents"] = (
    df["gross_total_cents"].astype(str)
)

df.loc[
    df["order_line_id"] == "line-0001",
    "gross_total_cents",
] = "2400.0"

df.to_csv(
    "incoming/decimal_amount.csv",
    index=False,
)
PY

Run the validator.

python quality.py \
    incoming/decimal_amount.csv \
    trusted/orders.csv

Output:

REJECTED: Line 2: invalid integer token in gross_total_cents: '2400.0'

The parser should reject the representation before constructing the fully typed financial dataframe.

The diagnostic identifies the offending column and token.

Why is this desirable if 2400.0 is mathematically equivalent to 2,400?

Because the contract describes more than numerical equivalence.

A change from integer cents to floating-point representations could indicate that a producer has changed its serialization or monetary calculation.

For critical financial pipelines, silently accepting a changed representation creates unnecessary ambiguity.

The validation boundary is the correct place to demand an explicit normalization decision.

12. Write a complete regression suite with pytest

A production-ready validation boundary needs more than a single successful example.

It should prove that known failure scenarios are actually rejected.

We’ll test:

• Valid order lines.

• Multiple lines sharing one parent order.

• Renamed columns.

• Changed gross-revenue meaning.

• String values in integer dataframe columns.

• Unexpected floating-point dtypes.

• Duplicate order-line identifiers.

• Unsupported contract versions.

• Unknown statuses.

• Empty batches.

• Unexpected columns.

• Malformed monetary representations.

• Duplicate CSV headers.

• Contract-defined limits.

• Preservation of trusted output after rejection.

• Successful publication after validation.

Create test_quality.py.

import pandas as pd
import pandera.pandas as pa
import pytest
import yaml

from quality import (
    CONTRACT_PATH,
    publish_orders,
    validate_orders,
)


@pytest.fixture
def valid_orders():
    return pd.DataFrame({
        "order_id": [
            "ord-0001",
            "ord-0001",
        ],
        "order_line_id": [
            "line-0001",
            "line-0002",
        ],
        "contract_version": [
            "v1",
            "v1",
        ],
        "quantity": [2, 1],
        "unit_price_cents": [1200, 350],
        "gross_total_cents": [2400, 350],
        "currency": ["USD", "USD"],
        "status": ["paid", "shipped"],
    })


def assert_rejected(
    df,
    contract_path=CONTRACT_PATH,
):
    with pytest.raises((
        ValueError,
        pa.errors.SchemaError,
        pa.errors.SchemaErrors,
    )):
        validate_orders(
            df,
            contract_path,
        )


def test_valid_orders_pass(valid_orders):
    assert len(
        validate_orders(valid_orders)
    ) == 2


def test_multiple_lines_can_share_order_id(
    valid_orders,
):
    assert (
        valid_orders["order_id"].nunique()
        == 1
    )

    assert len(
        validate_orders(valid_orders)
    ) == 2


def test_renamed_column_fails(valid_orders):
    drifted = valid_orders.rename(
        columns={
            "gross_total_cents": "net_total_cents"
        }
    )

    assert_rejected(drifted)


def test_semantic_drift_fails(valid_orders):
    drifted = valid_orders.copy()

    drifted.loc[
        0,
        "gross_total_cents",
    ] = 2000

    assert_rejected(drifted)


def test_numeric_string_in_dataframe_fails(
    valid_orders,
):
    drifted = valid_orders.copy().astype({
        "gross_total_cents": "object"
    })

    drifted.at[
        0,
        "gross_total_cents",
    ] = "2400"

    assert_rejected(drifted)


def test_decimal_dtype_in_dataframe_fails(
    valid_orders,
):
    drifted = valid_orders.copy()

    drifted["gross_total_cents"] = (
        drifted["gross_total_cents"].astype(float)
    )

    assert_rejected(drifted)


def test_duplicate_order_line_id_fails(
    valid_orders,
):
    drifted = valid_orders.copy()

    drifted.at[
        1,
        "order_line_id",
    ] = "line-0001"

    assert_rejected(drifted)


def test_unknown_version_fails(valid_orders):
    drifted = valid_orders.copy()

    drifted["contract_version"] = "v2"

    assert_rejected(drifted)


def test_unknown_status_fails(valid_orders):
    drifted = valid_orders.copy()

    drifted.at[
        0,
        "status",
    ] = "processing"

    assert_rejected(drifted)


def test_empty_batch_fails(valid_orders):
    assert_rejected(
        valid_orders.iloc[:0]
    )


def test_extra_column_fails(valid_orders):
    drifted = valid_orders.assign(
        discount_cents=400
    )

    assert_rejected(drifted)


@pytest.mark.parametrize(
    "bad_token",
    [
        "02400",
        "2,400",
        "NA",
        "2400.0",
        "",
        "$24.00",
    ],
)
def test_raw_amount_format_is_rejected(
    valid_orders,
    tmp_path,
    bad_token,
):
    source = tmp_path / "incoming.csv"
    destination = tmp_path / "trusted.csv"

    bad = valid_orders.copy().astype({
        "gross_total_cents": "object"
    })

    bad.at[
        0,
        "gross_total_cents",
    ] = bad_token

    bad.to_csv(
        source,
        index=False,
    )

    with pytest.raises(
        ValueError,
        match="invalid integer token",
    ):
        publish_orders(
            source,
            destination,
        )

    assert not destination.exists()


def test_duplicate_csv_header_fails(tmp_path):
    source = tmp_path / "bad.csv"

    source.write_text(
        "order_id,order_id\n"
        "ord-0001,ord-0002\n",
        encoding="utf-8",
    )

    with pytest.raises(
        ValueError,
        match="Duplicate column",
    ):
        publish_orders(
            source,
            tmp_path / "trusted.csv",
        )


def test_yaml_limits_drive_validation(
    valid_orders,
    tmp_path,
):
    contract = yaml.safe_load(
        CONTRACT_PATH.read_text(
            encoding="utf-8"
        )
    )

    contract["fields"]["quantity"]["max"] = 1

    custom_path = tmp_path / "contract.yaml"

    custom_path.write_text(
        yaml.safe_dump(contract),
        encoding="utf-8",
    )

    assert_rejected(
        valid_orders,
        custom_path,
    )


def test_failed_batch_preserves_trusted_file(
    valid_orders,
    tmp_path,
):
    source = tmp_path / "incoming.csv"
    destination = tmp_path / "trusted.csv"

    destination.write_text(
        "previous trusted snapshot\n",
        encoding="utf-8",
    )

    bad = valid_orders.copy()

    bad.at[
        0,
        "gross_total_cents",
    ] = 2000

    bad.to_csv(
        source,
        index=False,
    )

    with pytest.raises((
        pa.errors.SchemaError,
        pa.errors.SchemaErrors,
    )):
        publish_orders(
            source,
            destination,
        )

    assert destination.read_text(
        encoding="utf-8"
    ) == "previous trusted snapshot\n"


def test_valid_batch_is_published(
    valid_orders,
    tmp_path,
):
    source = tmp_path / "incoming.csv"
    destination = tmp_path / "trusted.csv"

    valid_orders.to_csv(
        source,
        index=False,
    )

    publish_orders(
        source,
        destination,
    )

    published = pd.read_csv(destination)

    assert len(published) == 2

    assert (
        published["gross_total_cents"].sum()
        == 2750
    )

Understanding the negative tests

Several tests validate individual columns.

Others test the complete publication process.

The difference matters.

A validator might correctly detect an invalid field while the surrounding ingestion application accidentally writes the batch anyway.

The test test_failed_batch_preserves_trusted_file verifies the operational safety property:

If validation fails, the previously trusted output must remain unchanged.

This test begins with an existing file, submits semantically invalid data, expects a Pandera rejection, and then verifies the original file content.

That is stronger than checking validation errors in isolation.

Testing the contract as the source of truth

The test test_yaml_limits_drive_validation changes the maximum quantity in a temporary copy of the YAML contract.

It then validates the same dataframe using the modified contract.

Because the Pandera schema is built from YAML, reducing the maximum quantity to one causes a previously valid row with quantity two to fail.

This demonstrates that the executable validator actually consumes the contract’s field limits.

The implementation also rejects unsupported rule names, which reduces the risk of silently ignoring a newly added YAML constraint.

These protections do not eliminate the need for code review, but they are a meaningful improvement over maintaining two independent sets of constants.

Testing CSV representation separately

The parameterized test for malformed amounts covers six input representations.

It includes a value with leading zeros, a thousands separator, a null marker, a decimal representation, an empty field, and a currency-formatted amount.

Each is rejected before publication.

This tests a different layer from Pandera’s dataframe dtype checks.

Together, the tests cover both the raw ingestion boundary and the typed dataframe boundary.

13. Run the complete validation suite

Execute the tests from the project root.

python -m pytest -q

The suite defines 16 test functions, one of which is parameterized with six cases, resulting in 21 collected test cases.

A successful run should report:

21 passed

This is the expected success summary, not a claim that the suite was executed against the specified environment here. The Python files have been checked for syntax, but a full Pandera 0.34.1 test run remains necessary before treating the implementation as release-verified.

To make that verification reproducible, keep the exact files and dependency manifest together, execute the command above, and retain the CI run as evidence.

A useful additional check is to print dependency versions before running pytest.

python -m pip show pandera pandas pytest PyYAML
python -m pytest -q

If a test fails, investigate the precise behavior rather than weakening a contract expectation just to obtain a green result.

14. Run the tests automatically in GitHub Actions

Validation logic should be reviewed and tested like application code.

Create .github/workflows/data-quality.yml.

The workflow below uses current GitHub Actions v7 releases pinned to specific commits: checkout v7.0.1 and setup-python v7.0.0. It also uses the committed requirements file for pip caching, eliminating the missing-dependency-file problem that can otherwise break the setup step.

name: Data Quality Contract Tests

on:
  push:
  pull_request:

permissions:
  contents: read

jobs:
  validate-contracts:
    runs-on: ubuntu-latest

    steps:
      - name: Check out repository
        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1
        with:
          persist-credentials: false

      - name: Set up Python
        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97
        with:
          python-version: "3.13"
          cache: pip
          cache-dependency-path: requirements.txt

      - name: Install dependencies
        run: |
          python -m pip install --upgrade pip
          python -m pip install -r requirements.txt

      - name: Display dependency versions
        run: |
          python -m pip show pandera pandas pytest PyYAML

      - name: Run contract regression tests
        run: |
          python -m pytest -q

The workflow performs four essential actions:

• Checks out the repository containing the contract and tests.

• Configures a supported Python runtime.

• Installs dependencies from the committed manifest.

• Executes the entire regression suite.

The explicit cache-dependency-path points to a file that actually exists in the project.

This is important because enabling pip caching without a matching dependency file can cause setup failures before any tests run.

The workflow uses read-only repository permissions and does not persist checkout credentials.

For a production repository, also require the validation job to pass before merging changes into protected branches.

CI and runtime validation serve different purposes

Continuous integration checks committed code and test fixtures.

Runtime validation checks actual incoming production data.

These controls should share the same implementation, but they are triggered at different moments.

Developer changes contract or validator
                  |
                  v
             Pull request
                  |
                  v
           CI regression tests
                  |
             Pass / Fail
                  |
                  v
          Reviewed deployment
                  |
                  v
         New production batch
                  |
                  v
        Runtime contract checks
                  |
             Pass / Fail
                  |
           +------+------+
           |             |
           v             v
        Publish       Quarantine

Passing CI does not guarantee that the next production batch will be valid.

A production system must execute the validator before allowing each new batch into the trusted dataset.

15. Define what happens when validation fails

Detecting bad data is only half the engineering problem.

The other half is deciding how the pipeline responds.

For critical financial datasets, silently continuing after a failed validation can be more damaging than temporarily delaying a report.

Our example adopts a fail-closed publication policy.

Critical failures should block publication

Examples include:

• Missing required fields.

• Duplicate order-line identifiers.

• Unknown contract versions.

• Invalid monetary values.

• Broken gross-total relationships.

• Unexpected schema changes.

• Invalid source representations.

• Empty batches when records are required.

These failures prevent publication until the data is corrected or an approved contract change is deployed.

However, retaining the old trusted dataset creates another consideration: freshness.

A previously validated snapshot may remain correct historically while becoming outdated for current operations.

Downstream systems should therefore distinguish between valid and fresh.

Preserve rejected data

The tutorial’s rejected input remains in the incoming directory, but a production system should use durable quarantine storage.

Each rejected batch should be associated with:

• Source system and batch identifier.

• Receipt and validation timestamps.

• Contract version.

• Validation failure details.

• Source schema.

• Responsible producer and consumer teams.

• Investigation status.

• Replay or remediation decision.

Do not overwrite the only copy of rejected data.

Quarantine is valuable because an upstream defect may require replaying the original event stream or reconstructing exactly which records were affected.

Separate warnings from blocking failures

Not every anomaly demands immediate rejection.

For example, a sudden increase in average order value might be a genuine commercial event.

A small deviation in daily order volume might be caused by seasonality or a promotional campaign.

These anomalies may initially trigger warnings, while violations of fundamental financial invariants block publication.

The policy should be explicit and associated with business impact.

A critical warning that nobody investigates is not meaningful protection.

16. Monitor quality characteristics that schemas cannot guarantee

Even a strong contract and comprehensive column checks leave gaps.

Several additional quality dimensions are particularly useful in production.

Freshness

A dataset may be structurally and semantically valid but too old to support operational decisions.

For each critical feed, define:

• Expected delivery frequency.

• Acceptable delay.

• Event-time versus ingestion-time semantics.

• Applicable time zone.

• Behavior during outages and backfills.

A data feed expected every 15 minutes should have an agreed lateness threshold rather than an arbitrary assumption that any delayed batch is corrupt.

Volume

A sudden drop in records may indicate an incomplete export or missing partition.

A sudden increase may indicate duplicate ingestion.

However, fixed row-count thresholds can generate false alarms during weekends, holidays, or seasonal peaks.

Volume expectations should reflect known business patterns.

Distribution

Suppose nearly all orders historically have quantities between one and five, but a new batch contains many quantities close to the maximum permitted value.

Every row might pass the schema, yet the distribution change could indicate an upstream bug.

Distribution checks can monitor:

• Order-value percentiles.

• Quantity distributions.

• Currency proportions.

• Status frequencies.

• Null rates.

• Category or product mixes.

Use these checks to identify unexplained shifts, not to assume that every unusual business event is invalid.

Independent reconciliation

Reconciliation is one of the strongest defenses against semantic changes that preserve internal consistency.

For example, compare gross order totals from the feed with totals independently calculated by the authoritative commerce service.

If both systems use the same broken transformation, the comparison may offer little protection.

The strongest reconciliation checks use sources with genuinely independent calculations or controls.

For financial metrics, specify exactly which amounts and periods are being compared.

17. Cherry on the cake: when real companies encountered silent data failures

The danger of valid-looking but incorrect information is not hypothetical.

Two 2026 accounts illustrate different aspects of this problem.

An unannounced source-field change distorted downstream metrics

In a June 16, 2026 Informatics Magazine interview, Olga Kokhan, CEO of Tinkogroup, described an incident in which a source field was modified without notice.

According to her firsthand account, the change did not stop the data pipeline from executing. Instead, it silently changed the meaning of downstream metrics.

The response was to give source-schema changes greater operational importance and monitor field structure, types, record volumes, and business metrics together.

The published account does not name the exact column or document its previous and new definitions. It therefore supports the narrower conclusion that an unannounced source-field modification altered downstream metric semantics—not a detailed reconstruction of a particular gross-to-net column change.

That distinction is important when learning from real incidents: the evidence should establish the behavior being discussed rather than invite invented technical details.

Nevertheless, the failure mode is remarkably close to our retail example.

A pipeline can remain operational while the values consumed by dashboards acquire a different interpretation.

Spotify’s upstream time-zone bug reached polished user-facing reports

A second example comes from Spotify’s March 12, 2026 engineering article, Inside the Archive: The Tech Behind Your 2025 Wrapped Highlights.

Spotify described a system generating personalized reports about notable days in listeners’ music histories.

During evaluation, engineers discovered that some reports were reporting incorrect discovery information.

The underlying issue was a subtle time-zone bug in an upstream data pipeline. The report-generation system then produced compelling narratives based on the incorrect day supplied by the pipeline.

Spotify’s engineers traced affected reports through metadata and evaluation records, fixed the upstream issue, removed affected output, and safely regenerated reports.

This was not an intentional column redefinition. It was an upstream temporal-semantics defect.

But the lesson is directly relevant.

A date value can be technically valid while representing the wrong business day.

For example, an event occurring near midnight may fall on different calendar dates depending on whether it is interpreted in UTC or a customer’s local time zone.

The data type alone cannot explain which interpretation is correct.

A robust daily-aggregation contract should define:

• The authoritative event timestamp.

• The time zone used for grouping.

• The day-boundary rule.

• How daylight-saving transitions are handled.

• How late-arriving events affect historical reports.

Tests should include boundary cases rather than relying only on ordinary daytime events.

What these incidents teach

Both cases demonstrate a general principle: successful downstream processing does not guarantee that upstream information retains its intended meaning.

A strong prevention strategy combines executable checks with clear ownership, independently verifiable business definitions, and a repair-and-replay process for affected outputs.

18. When to use Great Expectations instead of Pandera

Pandera works particularly well when the validation boundary is a Python dataframe.

Great Expectations is useful when teams need a broader framework for organizing expectation suites, batch validation, checkpoints, and validation results.

Its GX Core 1.x architecture provides explicit workflows for configuring data sources, defining expectations, and executing validation. See the Great Expectations documentation.

The concepts are similar, but their implementations differ.

For example, a Great Expectations suite could assert that:

• Required columns exist.

• Order-line identifiers are unique.

• Quantities fall within accepted bounds.

• Status values belong to an approved set.

• Critical monetary values are not null.

The organization would then execute those expectations against incoming batches.

One important principle remains unchanged.

A validation report is not a publication gate unless the surrounding pipeline acts on the validation result.

A system can generate excellent data quality documentation and still publish failed data if the workflow ignores unsuccessful validations.

Choose the tool according to architecture and operational requirements.

For a small Python-centered ingestion application, Pandera plus pytest may be sufficient.

For a platform coordinating many datasets, expectations, owners, and storage backends, Great Expectations may provide additional workflow capabilities.

The libraries are not mutually exclusive, and neither replaces a producer-consumer agreement.

19. Evolve contracts without breaking consumers

A data contract should make change safe, not prohibit change altogether.

As products evolve, fields are added, business definitions change, and new consumers appear.

The crucial requirement is that changes be deliberate and reviewable.

Additive changes

A producer introduces a new optional field, such as promotion_code.

This may be backward compatible if consumers are designed to ignore unknown optional fields.

Our tutorial uses strict schema enforcement, so it will reject even an added field until the contract changes.

That is an intentional conservative policy.

A production organization can adopt more flexible compatibility rules, but those rules should be documented and tested.

Structural breaking changes

A producer removes, renames, or changes the type of a required field.

For example, replacing gross_total_cents with net_total_cents is not automatically compatible.

The producer and consumer must agree whether the replacement preserves meaning or represents a new business quantity.

Changes may require coordinated deployment or temporary support for more than one schema version.

Semantic breaking changes

A producer changes the definition of a value without necessarily changing its shape.

Examples include:

• Gross revenue becoming net revenue.

• A UTC date becoming a local calendar date.

• Measurements switching from meters to centimeters.

• A status code acquiring a new business interpretation.

• Prices changing from tax-exclusive to tax-inclusive.

• Metrics switching from event-time to processing-time attribution.

These changes can be more dangerous than obvious structural breaks.

A semantic change should normally trigger explicit consumer review and, when appropriate, a new field name or contract version.

A safe migration process

A practical contract-evolution workflow looks like this:

1. The producer proposes the change and documents its semantics.

2. Consumers identify affected calculations and interfaces.

3. New or updated expectations are reviewed.

4. Regression tests verify both accepted and rejected cases.

5. Compatibility and deployment sequencing are agreed.

6. Historical backfills are planned where needed.

7. Runtime validation monitors the transition.

8. The previous contract is retired only after consumers migrate.

For breaking changes, a temporary dual-publishing strategy may be useful.

For example, a producer can publish both gross and net totals under explicitly different field names until consumers have updated their calculations.

This is safer than silently changing the contents of an existing field.

20. Production hardening beyond the tutorial

The example demonstrates a reliable local validation boundary, but larger systems need additional controls.

Concurrency and atomicity

Our use of os.replace() is appropriate for a local, single-writer example on a filesystem supporting atomic replacement.

It does not coordinate multiple publishers competing to update the same destination.

Production pipelines may require:

• Transactional database writes.

• Table-version commits.

• Compare-and-swap or conditional updates.

• Immutable object versions.

• A manifest pointing to the currently approved dataset.

For cloud object storage, do not assume a local-filesystem rename mechanism provides the same guarantees.

Large-dataset performance

Validation consumes resources.

Full uniqueness checks, cross-column comparisons, and reconciliation can become expensive on very large datasets.

Choose each check according to its required guarantee.

Sampling may be suitable for advisory distribution monitoring.

Sampling is not enough to prove complete uniqueness or the absence of invalid financial rows.

For strict guarantees, validate the complete relevant population or use storage-level constraints that enforce the same property.

Partitioning and global uniqueness

Our tutorial checks order-line uniqueness within one complete snapshot.

If an ingestion system receives multiple partitions or incremental batches, uniqueness within each batch does not guarantee uniqueness across the entire dataset.

In that architecture, enforce the key against the persisted dataset or against a suitable authoritative index.

The uniqueness scope must be part of the contract.

Replay and idempotency

A failed batch may be corrected and submitted again.

Production systems should associate batches with stable identifiers and avoid applying the same data twice.

An idempotent ingestion operation should produce the same logical result when the same approved batch is replayed.

This matters especially when retrying ingestion after network failures or partial system outages.

Protect contract integrity

A contract is security-sensitive configuration in the broader sense that it controls what data becomes trusted.

Store it in version control, review changes, and restrict unauthorized modifications.

An attacker or accidental change that weakens validation limits could allow previously rejected data into production.

Unknown or unsupported rules should fail clearly, and changes to critical constraints should require appropriate approvals.

21. Common implementation mistakes

Several mistakes repeatedly undermine data quality programs.

Mistake 1: Inferring the expected schema from every incoming batch.

A validator that treats the latest data as authoritative may adopt a producer’s mistake as the new expected format. Compare incoming data against an independently approved contract.

Mistake 2: Ignoring the row grain.

Requiring order_id to be unique in an order-line dataset would reject valid multi-line orders. Define keys at the correct grain.

Mistake 3: Assuming coerce=False protects raw input.

Earlier parsing stages may already have normalized malformed values. Validate representations before converting them where the wire format matters.

Mistake 4: Duplicating constraints across YAML and Python.

Independent definitions can disagree. Generate validators from supported contract rules or test for synchronization.

Mistake 5: Running dataframe-wide checks before verifying required columns.

A missing column can cause an incidental exception rather than a useful schema-drift diagnostic. Check structure before evaluating invariants.

Mistake 6: Writing only positive tests.

Accepting valid data does not prove the pipeline rejects invalid data. For each critical expectation, write at least one intentional violation.

Mistake 7: Reporting failures without enforcing them.

A monitoring dashboard that displays a failed expectation does not protect downstream consumers if the pipeline publishes the dataset anyway.

Mistake 8: Allowing CI and production rules to diverge.

Tests should exercise the same implementation used by ingestion.

Mistake 9: Silently accepting semantic changes.

A field can keep its name and type while acquiring a new business definition. Use business invariants, contract versions, change review, and reconciliation.

Mistake 10: Treating the previous trusted snapshot as permanently safe.

A rejected batch should not overwrite trusted data, but the retained snapshot may become stale. Report freshness separately.

22. A practical data quality checklist

Before deploying a validation system for an important dataset, review its guarantees.

Contract design

• The dataset has an identified producer and consumer.
• The row grain is explicitly documented.
• Primary keys and uniqueness scope are correct.
• Business meanings and measurement units are defined.
• Allowed values and numerical bounds are recorded.
• Contract versioning and change policies are documented.

Validation implementation

• Required columns are checked before dependent invariants.
• Unknown columns follow an explicit compatibility policy.
• Incoming types and representations are validated.
• Critical cross-column relationships are enforced.
• Empty, duplicate, malformed, and unsupported inputs are covered.
• Contract configuration and executable checks remain synchronized.

Regression and deployment

• Negative tests demonstrate rejection behavior.
• Tests use a committed dependency manifest or lockfile.
• CI runs before relevant changes are merged.
• Runtime validation executes before publication.
• Validation failures preserve previously trusted outputs.
• Affected inputs can be investigated and replayed.

Operations

• Rejected batches are quarantined durably.
• Alerts reach accountable owners.
• Dataset freshness is visible.
• Critical aggregates are reconciled independently.
• Semantic changes require consumer review.
• Historical backfills and migrations have an audit trail.

Not every dataset needs every check immediately.

The objective is to prioritize failures according to business impact and steadily expand automated coverage.

23. From reactive debugging to preventive data engineering

The most important improvement in this tutorial is not a particular Python library.

It is the architectural decision to make data correctness an enforced condition of publication.

We started with a retail order-line feed and documented its structure, grain, business semantics, and acceptable values.

We then built a contract-driven Pandera validator that checks raw source representations, dataframe types, identifiers, numerical constraints, and a meaningful financial invariant.

We deliberately introduced structural drift, changed the meaning of gross revenue, and supplied malformed monetary inputs.

Finally, we created regression tests for the behaviors that matter most: valid inputs should pass, invalid inputs should fail, and a rejected batch should not replace existing trusted data.

The central lesson is straightforward:

A successful pipeline is not necessarily a correct pipeline.

A validation suite cannot guarantee every aspect of correctness, but it can establish that data satisfies a defined, versioned, measurable set of promises.

Those promises should be owned by the teams producing and consuming the data. They should evolve through review, remain visible in documentation, and be enforced before bad information spreads.

Your next step: protect one critical dataset

Start with a dataset that matters to the business: an order feed, transaction export, sales report, inventory snapshot, or operational metrics table.

Document its five most important assumptions. Make them executable. Write negative tests for at least one structural change, one malformed input, and one semantic change.

Then place the validator immediately before publication.

Use the project in this tutorial as your starting point: run its regression suite in your pinned dependency environment, deliberately break the gross-total calculation, and confirm that rejected data leaves the trusted output unchanged.

Once the gate is working, integrate it into your production ingestion workflow and add monitoring for freshness, completeness, and independent reconciliation.

Make bad data fail at the door—not after it has reached three dashboards, two reports, and a business decision.