Data versioning and lineage: reproducing last month’s training set exactly

Imagine this production incident.

On October 16, 2026, your machine learning team discovers that a retail refund prediction model is producing suspicious results. The model was trained on September 30, evaluated successfully, and deployed shortly afterward.

The obvious debugging question is:

Can we reconstruct the exact training dataset used on September 30?

Unfortunately, the data warehouse has changed.

Historical orders have been corrected. Some refunds arrived late. Data pipelines have been updated. Feature definitions have evolved, and the current training query produces different records from the original run.

The original training dataset may no longer exist.

This is one of the most expensive reproducibility failures in production machine learning: the model artifact survives, but the evidence explaining how it was created does not.

Three engineering capabilities help prevent this situation:

• Data versioning preserves identifiable historical states of datasets.

• Data lineage records where data originated, which transformations produced it, and how downstream artifacts depend on it.

• Reproducibility combines data, code, configuration, and execution context to reconstruct or verify historical results.

In this tutorial, we will build a working dataset versioning system using Git, DVC 3.x, Python 3.14.8, and a local artifact store.

We will create September and October training datasets, introduce historical corrections, and recover September’s training set with byte-for-byte verification.

Then we will test deterministic regeneration, simulate artifact corruption, restore data from a fresh clone, examine lakehouse snapshots, and design a production-grade lineage contract.

Finally, we will examine a documented real-world training-data retention dispute that illustrates why preserving historical datasets matters beyond routine model debugging.

1. What does reproducing a training set exactly mean?

Before selecting tools, we need a precise definition of success.

Engineers use the word reproducible for several guarantees. These guarantees are related, but they are not interchangeable.

Level 1: Logical reproducibility

Running the same pipeline produces a dataset with equivalent logical records.

The records may be reordered. Floating-point values may differ slightly. Files may use different compression settings.

For some analytical purposes, this is sufficient.

It is not necessarily sufficient for investigating unexpected production predictions.

Level 2: Dataset-content reproducibility

The reconstructed dataset contains exactly the same:

• Training examples and their identifiers.

• Feature values and labels.

• Train, validation, and test membership.

• Relevant schema and feature semantics.

This is the most important level for many ML investigations.

Two Parquet files can have different binary hashes while representing identical logical rows. Therefore, logical dataset equality and file equality require different verification strategies.

Level 3: Byte-for-byte reproducibility

The reconstructed dataset contains precisely the same bytes as the original published artifact.

For a single file, the invariant is:

Original SHA-256 digest = Recovered SHA-256 digest

Matching cryptographic hashes provides strong evidence that the file contents are identical.

This detects changes caused by serialization differences, accidental overwrites, modified feature values, or altered record ordering.

Level 4: End-to-end training reproducibility

The system can also reconstruct the training execution.

That requires preserving:

• Exact datasets and labels.

• Feature transformation code.

• Train, validation, and test assignments.

• Model architecture and hyperparameters.

• Package and dependency versions.

• Random seeds and runtime configuration.

• Relevant hardware and execution characteristics.

Even with these details, identical model weights are not guaranteed. Some GPU operations and distributed training algorithms can be nondeterministic.

Our first goal is strict dataset reproducibility. We will then extend the design toward end-to-end reproducibility.

2. The four identifiers every training run needs

A common mistake is to use a date as a dataset version.

For example, a pipeline might refer to a dataset called orders_2026_09.

That name describes a period, not an immutable state.

If someone corrects a September transaction in October, the same query can return a different result while retaining the same dataset name.

A reliable training record needs four distinct identifiers.

The dependency graph looks like this:

Raw retail orders           source snapshot ID + content fingerprint
        |
        v
Feature construction        Git commit + configuration + runtime
        |
        v
Immutable training dataset  SHA-256 + schema + row count + split membership
        |
        v
Model training run          run ID + parameters + metrics + model artifact

Each transition represents a lineage relationship: an output depends on specific inputs and a recorded transformation.

This distinction matters because a Git commit alone is insufficient.

If the underlying dataset is stored outside Git and changes independently, Git may preserve the transformation code without preserving the original input records.

We need both.

3. Choosing a dataset versioning mechanism

Several technologies provide historical data recovery, but they work at different levels.

We will use DVC because its core workflow can run locally without provisioning a cloud platform.

DVC stores lightweight data references and pipeline definitions alongside Git history. Dataset contents live in a cache and can be published to an external remote.

When a historical Git revision is restored, DVC can retrieve the corresponding dataset contents.

For this tutorial, we will pin DVC to version 3.67.1, released March 31, 2026, as recorded on PyPI.

The same workflow can later use managed object storage instead of a local directory.

4. Hands-on project: two months of retail training data

Our example models an e-commerce problem: predicting whether an order will be refunded within seven days.

Each training record includes:

• Order ID and date.

• Store and product category.

• Quantity and unit price.

• Seven-day refund label.

There is an important temporal constraint.

An order placed on September 30 cannot safely receive a fully observed seven-day refund label on September 30.

Therefore:

• The September 30 snapshot includes eligible orders through September 23.

• The October 16 snapshot includes eligible orders through October 9.

We will also simulate corrections to older orders.

This matters because the October source dataset will not merely contain additional rows. Some historical records will have different feature values.

4.1. Set up the repository

This demonstration targets CPython 3.14.8, released September 30, 2026.

Python 3.10 reached end of life with its final release announced October 1, 2026. New production demonstrations should not depend on that unsupported series. See the official Python release announcement and Python 3.14.8 release notes.

The commands below assume:

• Python 3.14 installed as python3.14. Any 3.14.x patch release works; we re-ran the full lab with 3.14.6 and got the identical digests.

• Git installed.

• A Bash-compatible environment on Linux, macOS, or WSL.

Verify the interpreter before creating the environment.

python3.14 -c 'import sys; assert sys.version_info[:2] == (3, 14), sys.version'

Create the repository.

mkdir dataset-time-machine
cd dataset-time-machine

git init -b main

python3.14 -m venv .venv
source .venv/bin/activate

python -m pip install "dvc==3.67.1"

git config user.name "Dataset Demo"
git config user.email "demo@example.invalid"

printf ".venv/\n" > .gitignore

dvc init

mkdir -p scripts src config data/raw audit

dvc remote add -d archive ../versioned-store

The local DVC remote lives outside the Git repository.

This separation is essential.

Git preserves transformation code, pipeline configuration, and DVC metadata. The external directory preserves dataset objects.

A local remote demonstrates the mechanism, but it is not a durable backup if the underlying machine or disk fails.

In production, use storage with appropriate durability, access controls, and retention policies.

4.2. Generate deterministic retail orders

Create scripts/make_orders.py.

import argparse
import csv
from datetime import date, timedelta
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]
START = date(2026, 8, 1)

parser = argparse.ArgumentParser()
parser.add_argument(
    "--as-of",
    required=True,
    choices=["2026-09-30", "2026-10-16"],
)
args = parser.parse_args()

as_of = date.fromisoformat(args.as_of)

path = ROOT / "data/raw/orders.csv"
path.parent.mkdir(parents=True, exist_ok=True)

fields = [
    "order_id",
    "order_date",
    "store_id",
    "category",
    "quantity",
    "unit_price_cents",
    "refunded_within_7d",
]

categories = ["books", "home", "games", "office"]

days = (as_of - START).days + 1

with path.open(
    "w",
    encoding="utf-8",
    newline="",
) as file:
    writer = csv.DictWriter(
        file,
        fieldnames=fields,
        lineterminator="\n",
    )
    writer.writeheader()

    for index in range(days * 5):
        order_day = START + timedelta(
            days=index // 5
        )

        price = 1200 + (index * 137) % 9000

        if (
            args.as_of == "2026-10-16"
            and index in {13, 77, 140}
        ):
            price += 250

        writer.writerow(
            {
                "order_id": f"O{100000 + index}",
                "order_date": order_day.isoformat(),
                "store_id": f"S{index % 12 + 1:02d}",
                "category": categories[
                    index % len(categories)
                ],
                "quantity": 1 + index % 3,
                "unit_price_cents": price,
                "refunded_within_7d": int(
                    index % 23 == 0
                ),
            }
        )

print(f"Generated {days * 5} orders for {as_of}")

This generator deliberately avoids network access, external datasets, and randomness.

The experiment remains controlled because every change comes from the selected snapshot date and our explicit historical corrections.

Three orders receive revised prices in the October dataset.

In a production system, those corrections might arrive through a change data capture stream, a warehouse update, or a corrected upstream export.

Our generator simulates the resulting source state rather than the ingestion system.

4.3. Build the training dataset

Create src/build_training_set.py.

The transformation will:

1. Load the raw orders.

2. Apply the label maturity cutoff.

3. Generate a deterministically ordered training file.

4. Write a manifest containing provenance and integrity information.

import csv
import hashlib
import json
from datetime import date, timedelta
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]

source = ROOT / "data/raw/orders.csv"
output = ROOT / "data/train.csv"
manifest = ROOT / "data/train.manifest.json"

config = json.loads(
    (ROOT / "config/train.json").read_text(
        encoding="utf-8"
    )
)

as_of = date.fromisoformat(config["as_of_date"])
label_delay = int(config["label_delay_days"])

if label_delay < 0:
    raise ValueError(
        "label_delay_days must be nonnegative"
    )

eligible_through = as_of - timedelta(
    days=label_delay
)

with source.open(
    "r",
    encoding="utf-8",
    newline="",
) as file:
    source_rows = list(csv.DictReader(file))

rows = []

for record in source_rows:
    order_day = date.fromisoformat(
        record["order_date"]
    )

    if order_day > as_of:
        raise ValueError(
            "Source contains orders after as_of_date"
        )

    if order_day <= eligible_through:
        rows.append(
            {
                "order_id": record["order_id"],
                "order_date": record["order_date"],
                "store_id": record["store_id"],
                "category": record["category"],
                "quantity": record["quantity"],
                "unit_price_cents": record[
                    "unit_price_cents"
                ],
                "label_refunded_7d": record[
                    "refunded_within_7d"
                ],
            }
        )

rows.sort(key=lambda record: record["order_id"])

if (
    not rows
    or len({row["order_id"] for row in rows})
    != len(rows)
):
    raise ValueError(
        "Training set is empty or has duplicate order IDs"
    )

fields = [
    "order_id",
    "order_date",
    "store_id",
    "category",
    "quantity",
    "unit_price_cents",
    "label_refunded_7d",
]

with output.open(
    "w",
    encoding="utf-8",
    newline="",
) as file:
    writer = csv.DictWriter(
        file,
        fieldnames=fields,
        lineterminator="\n",
    )
    writer.writeheader()
    writer.writerows(rows)


def sha256(path):
    return hashlib.sha256(
        path.read_bytes()
    ).hexdigest()


summary = {
    "as_of_date": as_of.isoformat(),
    "eligible_through": eligible_through.isoformat(),
    "label_delay_days": label_delay,
    "row_count": len(rows),
    "source_sha256": sha256(source),
    "training_sha256": sha256(output),
}

manifest.write_text(
    json.dumps(
        summary,
        indent=2,
        sort_keys=True,
    ) + "\n",
    encoding="utf-8",
)

print(
    f"Built {len(rows)} training rows; "
    f"SHA-256: {summary['training_sha256']}"
)

Several implementation details are important.

Stable ordering. Rows are sorted by order ID. Without deterministic ordering, equivalent datasets can produce different file hashes.

Stable serialization. UTF-8 encoding and Unix newline characters are selected explicitly.

Explicit label cutoff. The observation window is configuration, not an implicit assumption.

Content fingerprints. Both the source file and generated training file receive SHA-256 digests.

Quality gates. The script rejects empty results, duplicate order IDs, and source records dated after the declared snapshot.

This implementation loads the source into memory because the synthetic dataset is small. A production pipeline processing terabytes would require streaming or distributed execution.

One additional caveat: our synthetic source already contains final refund labels. A real historical training pipeline must derive those labels from events observable by the relevant cutoff, not from today’s corrected outcome data.

Otherwise, the pipeline could reproduce a file perfectly while still leaking future information into model training.

4.4. Define the September configuration

Create config/train.json.

{
  "as_of_date": "2026-09-30",
  "label_delay_days": 7
}

These two fields represent different business concepts.

The snapshot date identifies the source state represented by the run.

The label delay determines when an order becomes eligible to contribute a fully observed training label.

Keeping them separate prevents an easy mistake: assuming that because an order exists, its outcome is already known.

4.5. Declare the DVC lineage graph

Create dvc.yaml in the repository root.

stages:
  build_training_set:
    cmd: python src/build_training_set.py
    deps:
      - src/build_training_set.py
      - config/train.json
      - data/raw/orders.csv
    outs:
      - data/train.csv
      - data/train.manifest.json

This file declares the training dataset’s direct dependencies.

DVC uses the stage definition to determine what needs to be executed. It also maintains dvc.lock, which records resolved dependency and output hashes.

The important concept is that lineage becomes executable.

If the source file, transformation code, or configuration changes, DVC can identify that the existing outputs may no longer correspond to their recorded inputs.

4.6. Create a reusable integrity verifier

Instead of manually writing one-off verification scripts after each recovery, we will create a reusable program.

Create scripts/verify_dataset.py.

import argparse
import csv
import hashlib
import json
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]

parser = argparse.ArgumentParser()
parser.add_argument("--expected", required=True)
parser.add_argument("--rows", type=int, required=True)
parser.add_argument("--as-of", required=True)
args = parser.parse_args()

training = ROOT / "data/train.csv"
source = ROOT / "data/raw/orders.csv"
manifest_path = ROOT / "data/train.manifest.json"

expected_path = ROOT / args.expected


def sha256(path):
    digest = hashlib.sha256()

    with path.open("rb") as file:
        for chunk in iter(
            lambda: file.read(1024 * 1024),
            b"",
        ):
            digest.update(chunk)

    return digest.hexdigest()


expected = expected_path.read_text(
    encoding="utf-8"
).strip()

actual = sha256(training)

manifest = json.loads(
    manifest_path.read_text(
        encoding="utf-8"
    )
)

with training.open(
    "r",
    encoding="utf-8",
    newline="",
) as file:
    row_count = sum(
        1 for _ in csv.DictReader(file)
    )

checks = {
    "expected_digest": actual == expected,
    "manifest_digest": (
        actual == manifest["training_sha256"]
    ),
    "source_digest": (
        sha256(source) == manifest["source_sha256"]
    ),
    "expected_rows": row_count == args.rows,
    "manifest_rows": (
        row_count == manifest["row_count"]
    ),
    "snapshot_date": (
        manifest["as_of_date"] == args.as_of
    ),
}

failed = [
    name
    for name, passed in checks.items()
    if not passed
]

if failed:
    raise SystemExit(
        "Dataset verification failed: "
        + ", ".join(failed)
    )

print("Dataset integrity verified")
print("SHA-256:", actual)
print("Rows:", row_count)
print("As-of:", manifest["as_of_date"])

This verifies more than the final training hash.

It checks:

• The independent reference digest.

• The digest embedded in the manifest.

• The raw source digest.

• The actual and declared row counts.

• The expected historical snapshot date.

The verifier is intentionally separate from the transformation script. That separation makes it useful for validating both restored artifacts and freshly regenerated output.

5. Publish the September 2026 training set

We can now create our first historical state.

Run:

python scripts/make_orders.py \
  --as-of 2026-09-30

dvc add data/raw/orders.csv

dvc repro

dvc push

The generator produces 305 raw orders.

The training transformation accepts 270 orders eligible through September 23.

The expected training file SHA-256 digest for the exact code above is:

9eb30129b4755d425927c0894362c7178d1b33e1a4397fcf0eb3362519f3c2d3

Record this digest independently in Git-tracked audit metadata.

python - <<'PY'
import hashlib
from pathlib import Path

path = Path("data/train.csv")

digest = hashlib.sha256(
    path.read_bytes()
).hexdigest()

Path("audit/2026-09.sha256").write_text(
    digest + "\n",
    encoding="utf-8",
)

print(digest)
PY

The heredoc is a Bash command containing embedded Python, so the entire snippet correctly uses the bash language tag.

Now verify the dataset.

python scripts/verify_dataset.py \
  --expected audit/2026-09.sha256 \
  --rows 270 \
  --as-of 2026-09-30

Commit the historical state.

git add .
git commit -m "Publish September 2026 training set"

git tag training-2026-09

git rev-parse HEAD

The final command prints the full Git commit SHA. In a production training record, retain that full identifier alongside the human-readable tag.

At this point, September’s dataset has three different references:

1. A Git revision containing transformation code, configuration, DVC metadata, and the audit digest.

2. Content-addressed DVC objects published to the external artifact store.

3. A SHA-256 digest identifying the exact training file.

The Git tag provides a convenient release name.

The DVC metadata identifies which data objects belong to the historical revision.

The external artifact store preserves the actual bytes.

This is why committing code to Git is not a substitute for publishing data to DVC storage.

Both histories must survive.

6. Introduce October changes and publish a second version

Now imagine the normal evolution of a retail business.

New orders have arrived.

Three historical orders have been corrected.

The label maturity window has advanced.

A new model training run should consume a new dataset.

Update the configuration using an executable Bash heredoc.

python - <<'PY'
import json
from pathlib import Path

path = Path("config/train.json")

config = json.loads(
    path.read_text(encoding="utf-8")
)

config["as_of_date"] = "2026-10-16"

path.write_text(
    json.dumps(config, indent=2) + "\n",
    encoding="utf-8",
)
PY

Generate the October source state.

python scripts/make_orders.py \
  --as-of 2026-10-16

dvc add data/raw/orders.csv

dvc repro

dvc push

Commit the new version.

git add .
git commit -m "Publish October 2026 training set"

git tag training-2026-10

The October generator produces 385 raw orders and 350 eligible training records.

Because historical prices and the cutoff have changed, the output fingerprint differs from September’s.

The October training file SHA-256 is:

fb5db078d241eb6fdc1aa066e0c48b3912bcfcc3fc9eca3cc8fe284342ef7e23

Compare the two dataset versions.

These values are deterministic outputs of the synthetic generator.

The key observation is that October does not simply overwrite September’s historical artifact in DVC storage.

The working filename is still data/train.csv.

But the stored content identity has changed.

A filename describes where the current working copy appears. It does not identify a particular historical version.

7. Recover last month’s training set exactly

Now comes the important test.

We have the October dataset in our working directory.

We want to recover the exact training file consumed by the September 30 run, even though the raw source has since changed.

7.1. Restore the historical Git revision

Run:

git switch --detach training-2026-09

Git restores the historical tracked files, including the September transformation script, configuration, DVC pointers, lock metadata, and audit digest.

The detached HEAD is intentional. We are inspecting a historical release rather than modifying the current development branch.

However, switching Git revisions does not automatically restore all DVC-managed data files.

Those must be materialized separately.

7.2. Restore the historical dataset

Run:

dvc pull

DVC retrieves the objects referenced by the checked-out metadata and materializes them in the workspace.

The command effectively combines fetching remote objects into the DVC cache and checking out the corresponding workspace files.

This is the behavior described in the official DVC pull documentation.

If DVC encounters local modifications that would be removed or replaced, it may request confirmation.

For a controlled recovery where you intentionally want the published historical state, DVC provides a force option:

dvc pull --force

Warning: Use this option only when you are willing to discard conflicting local workspace changes. Preserve any work you need before forcing restoration.

The crucial property is that the original September bytes are retrieved, not recreated from October’s current source.

7.3. Verify the original training file

Execute the reusable verifier.

python scripts/verify_dataset.py \
  --expected audit/2026-09.sha256 \
  --rows 270 \
  --as-of 2026-09-30

Expected output:

Dataset integrity verified
SHA-256: 9eb30129b4755d425927c0894362c7178d1b33e1a4397fcf0eb3362519f3c2d3
Rows: 270
As-of: 2026-09-30

This test is stronger than checking filenames, timestamps, or row counts.

It demonstrates that:

• The recovered training file matches its committed reference digest.

• The manifest agrees with the recovered file.

• The raw input matches its historical fingerprint.

• The expected September row count and snapshot date are preserved.

We have recovered the exact published training artifact.

The recovery depends on DVC still retaining the referenced objects, not on the current upstream source being correct.

7.4. Independently test deterministic regeneration

Restoring an artifact and regenerating it are two different capabilities.

Artifact restoration asks:

Can I retrieve the bytes originally published?

Deterministic regeneration asks:

Can I run the historical transformation again and obtain those same bytes?

We should test both.

Run:

dvc repro --force --no-run-cache

python scripts/verify_dataset.py \
  --expected audit/2026-09.sha256 \
  --rows 270 \
  --as-of 2026-09-30

dvc status

The --force option tells DVC to execute the stage even when dependencies have not changed.

The additional --no-run-cache option ensures this test executes the stage command instead of satisfying it through a previously recorded run result.

That distinction matters for reproducibility testing. We want to test the transformation itself, not merely retrieve another cached output.

The verifier runs immediately after regeneration.

If it succeeds, we have independently established that this deterministic pipeline can recreate the exact September file from the preserved September inputs.

If it fails, investigate source data, code, runtime, serialization, or undeclared dependencies before claiming full reproducibility.

Finally, return to the development branch.

git switch main

dvc pull

Our primary guarantee is now established:

Given the September Git revision and retained DVC objects, the original September training dataset can be recovered and verified byte for byte.

8. Prove recovery works with an empty local cache

The previous test restored historical data in a workspace that already had access to previously downloaded DVC objects.

That is useful, but it is not sufficient evidence of durable remote storage.

A developer’s local cache could contain objects that have already been deleted from the configured remote.

To test actual recovery, use a fresh Git clone with its own empty DVC cache.

8.1. Understand the relative remote-path problem

Our original remote was configured as:

dvc remote add -d archive ../versioned-store

That relative path is convenient for a local tutorial.

However, a clone in another directory or on another machine may not have a corresponding ../versioned-store directory.

The Git revision can be correct while the remote URL points to a nonexistent location.

Therefore, a recovery environment must explicitly configure how to reach the artifact store.

8.2. Create an independent clone

From the original repository, return to its parent directory.

cd ..

ARCHIVE="$(pwd)/versioned-store"

git clone dataset-time-machine dataset-time-machine-cold

cd dataset-time-machine-cold

python3.14 -m venv .venv
source .venv/bin/activate

python -m pip install "dvc==3.67.1"

This new repository has its own DVC cache.

It does not inherit the original repository’s cached training artifacts.

8.3. Configure the artifact store explicitly

Override the remote URL for this clone.

dvc remote modify --local archive url "$ARCHIVE"

dvc remote list

The --local flag writes the override to .dvc/config.local.

That file is Git-ignored and is intended for environment-specific settings.

The original project’s shared remote configuration remains unchanged.

See the DVC remote modification documentation for details.

In a real deployment, this override would typically point to a persistent object-storage endpoint rather than another local directory.

8.4. Recover September without the original cache

Run:

git switch --detach training-2026-09

dvc pull

python scripts/verify_dataset.py \
  --expected audit/2026-09.sha256 \
  --rows 270 \
  --as-of 2026-09-30

If verification succeeds, the fresh clone has demonstrated something stronger than the original workspace test.

It successfully retrieved historical data from the configured artifact store without relying on the original repository’s local cache.

Because our tutorial remote is still on the same filesystem, this does not prove disaster recovery from machine failure.

But it does prove that an independent workspace can resolve and retrieve the historical data objects.

For a production recovery drill, place the remote in durable external storage and repeat the same procedure from an isolated execution environment.

9. What DVC captures—and what it does not

DVC provides versioned artifacts and an executable dependency graph.

It does not automatically discover every dependency a pipeline might have.

Suppose our transformation script also reads an external pricing API.

If the API response is not preserved or associated with an immutable version, running identical code later may produce different features.

The same problem appears with:

• Mutable feature-store lookups.

• Queries against unversioned database tables.

• External enrichment services.

• Runtime-generated timestamps.

• Configuration loaded outside the repository.

• Untracked package or environment changes.

A useful engineering principle is:

Reproducible output = function of recorded inputs, code, configuration, and execution semantics

If an important input is missing from that record, the lineage is incomplete.

Data lineage must cover the full production graph

Consider a real retail ML system.

Orders (source snapshot) ----+
                             +--> point-in-time joins and feature generation
Refund events (source snapshot)       (transformation revision + temporal rules)
                                              |
                                              v
                                Versioned training dataset
                                (fingerprint + schema + split membership)
                                              |
                                              v
                                Registered model
                                (training run ID + artifact digest)

Each source needs an independently identifiable version.

A fingerprinted final training CSV is valuable, but it does not necessarily reveal whether a wrong label originated from the order platform, a refund event feed, a defective temporal join, or an outdated feature definition.

That requires upstream lineage.

10. The point-in-time correctness problem

One of the hardest reproducibility problems is distinguishing when something happened from when a system learned about it.

Consider an order placed on September 18.

A price correction is entered into the source system on October 4.

The corrected record might still carry the original order date of September 18.

If a reconstruction query filters only by order date, it may incorrectly include October’s correction in the September 30 training state.

This is a temporal correctness failure.

Business time versus observation time

Production data frequently needs two temporal dimensions.

Valid time describes when a fact applies in the business domain.

Observation or system time describes when the data platform learned or recorded that fact.

For example:

For a September 30 reconstruction, the second record was not yet observable and must not replace the original value.

For an October 16 reconstruction, the correction may be applicable.

A bitemporal data model preserves these dimensions separately.

Conceptually, reconstruction requires selecting the latest applicable source version that was observable no later than the historical cutoff, while also applying the relevant business-time rules.

Filtering today’s mutable table by historical event dates does not accomplish this.

Why immutable snapshots are valuable

With a preserved September source snapshot, we do not need to infer what a database knew on September 30.

We can directly recover its recorded historical state.

However, it still matters where in the pipeline the snapshot was established.

A snapshot taken after an incorrect feature join faithfully preserves the incorrect output.

Versioning creates evidence. It does not guarantee that the evidence represents correct business logic.

11. What changes when datasets become terabytes?

Our DVC demonstration works well for standalone files.

Enterprise training inputs, however, may live in warehouse tables or object-storage lakehouses containing billions of records.

At that scale, repeatedly copying entire datasets can become costly.

Two architectural patterns are particularly useful.

Option A: lakeFS for object-storage versioning

lakeFS brings version-control concepts to object-storage data.

It supports commits, branches, and historical references to object collections.

Its current versioning documentation explains how data changes can be committed without duplicating all underlying objects.

For machine learning workflows, this enables:

• Establishing a stable input state before training.

• Isolating ingestion and preprocessing experiments.

• Comparing historical versions of object collections.

• Promoting validated data changes.

• Associating training runs with immutable commits.

The important design choice is to record a resolved commit ID, not merely a mutable branch reference.

A branch name tells you which state is currently selected.

An immutable commit identifies a particular historical state.

Option B: Apache Iceberg snapshots

Apache Iceberg provides transactional analytical tables with snapshot-based history.

Its metadata records snapshots that support reader isolation and time-travel queries.

Iceberg also supports branches and tags for managing historical references.

The official Iceberg branching and tagging documentation describes using historical tags to preserve snapshots for auditing.

This is valuable when training inputs originate from large analytical tables.

However, snapshot IDs alone do not guarantee permanent recoverability.

Iceberg supports snapshot expiration and cleanup of data files no longer referenced by retained snapshots.

Therefore, historical snapshots must remain protected by appropriate retention settings.

Imagine an organization retains production models for eighteen months but expires all unprotected historical source snapshots much sooner.

The model may remain available long after the evidence needed to reconstruct its training data disappears.

That is a retention-policy mismatch.

Choosing the appropriate layer

These technologies are complementary.

An enterprise pipeline might use Iceberg for raw and feature tables, DVC for exported training artifacts, and an experiment tracker for model training runs.

The essential requirement is to connect their immutable identifiers.

12. Design a production-grade lineage manifest

Our tutorial writes a compact manifest containing:

• Snapshot date.

• Label maturity window.

• Row count.

• Source SHA-256.

• Training dataset SHA-256.

A mature production platform should record considerably more information.

The following is an illustrative contract. It is not metadata generated automatically by our demo, and symbolic execution fields must be populated from the actual training environment before publication.

{
  "manifest_schema_version": 1,
  "run_id": "refund-model-2026-09-30-001",
  "created_at_utc": "2026-09-30T18:00:00Z",
  "data": {
    "source_system": "retail-orders",
    "source_snapshot_id": "orders-snapshot-20260930-v1",
    "training_artifact": "data/train.csv",
    "row_count": 270,
    "sha256": "9eb30129b4755d425927c0894362c7178d1b33e1a4397fcf0eb3362519f3c2d3",
    "schema_version": "refund-features-v1"
  },
  "features": {
    "as_of_date": "2026-09-30",
    "label_delay_days": 7,
    "eligible_through": "2026-09-23"
  },
  "execution": {
    "git_revision": "RESOLVE_FULL_GIT_COMMIT_SHA",
    "pipeline_stage": "build_training_set",
    "python_version": "3.14.8",
    "dvc_version": "3.67.1",
    "dependency_lock": "requirements.lock"
  },
  "validation": {
    "dataset_hash_verified": true,
    "unique_order_ids": true,
    "label_maturity_verified": true
  }
}

In production, the dependency_lock field must point to a real, retained dependency specification.

The full Git revision must identify the actual transformation code.

For stronger guarantees, the manifest should also reference a container image digest, relevant system libraries, training parameters, data quality reports, and model artifact fingerprint.

Record splits, not just random seeds

Recording a random seed helps, but is not always enough.

Dataset splitting can change when:

• Input ordering changes.

• New rows are inserted.

• Library implementations change.

• Group membership changes.

• Time-based cutoff rules change.

• Preprocessing excludes different records.

For strict reproducibility, persist the actual membership of the training, validation, and test sets.

One option is to store immutable example identifiers and split assignments together.

Another is to publish separately fingerprinted split artifacts.

This allows investigators to identify precisely which examples were used for model fitting and which were reserved for evaluation.

Preserve schema semantics

Column names alone do not establish feature meaning.

A robust schema contract should describe:

• Logical and physical data types.

• Nullability and valid ranges.

• Units and categorical encodings.

• Missing-value handling.

• Feature-generation rules.

• Target definition and observation window.

For example, a price column measured in cents is not equivalent to a price column measured in major currency units.

Two datasets can have identical column names and row counts while representing different model inputs.

Capture the exact execution environment

A reproducibility contract should record enough runtime information to explain changes in computation.

Important details may include:

• Python patch version.

• Dependency lockfile.

• Container image digest.

• Numerical library versions.

• CPU or GPU execution mode.

• Determinism configuration.

• Distributed training configuration.

Dataset byte-for-byte equality is often achievable with careful serialization.

Model-weight byte-for-byte equality is a separate and more demanding objective.

13. The restoration test your CI should run

Versioning is only valuable when recovery actually works.

Production teams should periodically restore historical training runs from clean environments.

A useful CI or operational test should select an older production model and verify all of the following.

Historical training-run recovery checklist

• The historical Git commit exists and can be checked out.
• Every referenced dataset object is retrievable from persistent storage.
• The recovered training artifact matches its recorded SHA-256 digest.
• Historical row-count and schema validations pass.
• Source, code, and configuration references agree.
• Original train, validation, and test assignments are recoverable.
• Replaying the historical pipeline preserves the expected dataset.
• Artifact retention and access controls protect required historical evidence.

The test must target historical versions, not just the latest successful training run.

A recently completed pipeline can appear healthy while older DVC objects have been garbage-collected or Iceberg snapshots have expired.

For important production models, conduct the test from an isolated environment with an empty local data cache.

That verifies actual artifact-store recoverability rather than accidentally depending on objects retained on a developer’s machine.

14. Cherry on the cake: a real training-data deletion dispute

A documented real-world example illustrates why historical training-data retention matters beyond routine ML debugging.

The case involves OpenAI’s historical Books1 and Books2 training datasets (originally named “LibGen1” and “LibGen2”, per the court) and the copyright litigation over the data used to train its language models (In re OpenAI, Inc. Copyright Infringement Litigation, S.D.N.Y.).

The datasets were deleted in 2022. At the time, OpenAI gave “non-use” as the reason. Once the deletion surfaced in litigation, why the training data had been deleted became a discovery fight of its own. In a November 24, 2025 opinion and order, Magistrate Judge Ona Wang noted that OpenAI’s position on whether those reasons were privileged had shifted several times. She ordered OpenAI to produce internal communications, including with in-house lawyers, about the deletion.

What this incident proves—and what it does not

The documented facts show a genuine, high-stakes dispute about historical training data that no longer existed in its original form. Years later, nobody could simply check out “the training set as of 2021”.

They do not show that a specific technical investigation was blocked, and the litigation is still ongoing.

That distinction matters.

Our opening retail-model failure is a hypothetical engineering incident.

The court case is a verified real-world example of how much effort follows when a historical training dataset was deleted instead of versioned.

The engineering connection is direct.

Suppose a model’s training data is removed after deployment.

Investigators later need to determine:

• Which source documents or records were included?

• Which versions of those records were used?

• Can the historical data be recovered completely?

• Is a recovered dataset identical to the original?

• Can independent parties verify those claims?

Without preserved content identities and complete historical artifacts, answering these questions becomes harder.

A recovered approximation is not automatically equivalent to the original training dataset.

Even if a reconstruction contains many of the same records, a small difference in examples, labels, preprocessing, or membership may matter materially to an investigation.

The deeper lesson: preserve evidence before you need it

There are three distinct capabilities.

Retention: The original data remains stored somewhere.

Recoverability: An authorized investigator can retrieve the correct historical version.

Verifiability: The investigator can establish that the recovered bytes or logical records match the original published state.

An organization can satisfy one capability while failing another.

A cryptographic hash, for example, provides a content fingerprint, but it cannot recreate missing records.

A storage backup may preserve files without clearly identifying which ones correspond to a particular training run.

And a reconstructed dataset may look plausible without being identical to the original.

The full reproducibility contract requires all three capabilities to work together.

15. What happens when the original data really is gone?

Consider a separate hypothetical failure.

A model was trained from a warehouse query in September.

The team committed its Python training script but did not record a source snapshot, export the exact training dataset, or preserve the original split assignments.

In October, historical source records were corrected and older database states were removed.

The trained model artifact still exists.

Can the September training set be reconstructed exactly?

Not necessarily.

The query and code describe how data was selected.

They do not recreate historical facts that the source system no longer retains.

Investigators could attempt to recover those facts from:

• Database backups.

• Change data capture archives.

• Warehouse audit logs.

• Earlier data exports.

• Feature-store history.

• Cached training artifacts.

But if none of these systems retains sufficient information to reconstruct the original state, exact recovery is impossible.

A hash cannot repair this situation.

It can establish whether a candidate dataset matches a recorded reference.

It cannot regenerate the missing dataset from its digest.

This is the fundamental distinction between integrity verification and recoverability.

16. Common mistakes that destroy reproducibility

Sophisticated data platforms can still fail when their historical identity and retention contracts are incomplete.

Mistake 1: Recording a mutable path instead of a version

An object-storage path, table name, or branch reference may resolve to different data over time.

Record immutable commits, object versions, snapshot IDs, or artifact fingerprints.

Mistake 2: Versioning raw data but not generated features

Raw inputs may remain stable while feature definitions and enrichment data change.

Preserve the final training artifact whenever practical, alongside its dependencies.

Mistake 3: Treating timestamps as snapshot identifiers

A date filter is not equivalent to a historical database snapshot.

Late-arriving events, corrections, and backfills can change the result of a time-filtered query.

Use immutable snapshots or explicit bitemporal reconstruction rules.

Mistake 4: Allowing retention policies to remove required evidence

Production model retention and historical data retention must be coordinated.

For DVC, protect required remote objects from inappropriate garbage collection or storage lifecycle deletion.

For Iceberg, ensure snapshot and reference retention settings preserve required history.

A pointer to deleted content is not a usable historical version.

Mistake 5: Comparing only row counts

Two datasets can have identical row counts while containing different feature values, labels, or assignments.

Use content hashes for exact-file equality and record-level comparisons when logical equivalence is the intended guarantee.

Mistake 6: Ignoring serialization nondeterminism

Parallel writers, unstable row ordering, compression metadata, runtime timestamps, and floating-point formatting can change file bytes without changing logical records.

Define whether your contract requires identical bytes or equivalent data.

Then build tests appropriate to that contract.

Mistake 7: Assuming a hash proves authenticity

SHA-256 strongly identifies file contents.

It does not independently prove who produced the digest or whether the recorded digest itself has been maliciously modified.

For stronger integrity assurance, use protected manifests, authenticated publication, signed attestations where appropriate, and access-controlled storage.

Mistake 8: Confusing dataset reproducibility with model reproducibility

Recovering the exact training CSV does not guarantee identical model parameters.

Full training reproduction may require container image digests, pinned dependencies, deterministic algorithms, hardware configuration, and distributed runtime details.

17. Build a production reference architecture

The complete pattern can be implemented as a series of controlled publication steps.

Stage A: Source publication

Ingest incoming source changes into a version-capable storage layer.

Validate data quality.

Establish an immutable reference to the accepted source state.

Publish metadata that identifies the source snapshot and its integrity properties.

Stage B: Feature publication

Run feature transformations against pinned source versions.

Enforce point-in-time correctness and label maturity.

Record feature definitions and schema semantics.

Persist dataset membership and split assignments.

Publish the generated training dataset with an integrity manifest.

Stage C: Training publication

Record the training artifact identity.

Capture the transformation revision and runtime environment.

Associate the experiment run with its complete input manifest.

Store the resulting model artifact with a content fingerprint.

Stage D: Release and retention

Promote the validated model and lineage manifest together.

Protect required historical objects against unintended modification or deletion.

Align artifact retention with the model’s investigation and compliance requirements.

Schedule restoration tests from independent environments.

This architecture does not require one universal platform.

It requires a reliable contract between the source data platform, feature pipeline, training system, model registry, and artifact store.

The contract should answer three questions for every production model:

What exact data created this model?

Where is that historical data retained?

Can an independent environment retrieve and verify it today?

If any answer depends on someone remembering which notebook or warehouse table they used, the lineage system is incomplete.

18. Practical exercise: corrupt and restore a historical artifact

We will finish with a deliberate integrity failure.

This exercise demonstrates why preserved digests and reliable restoration procedures are valuable.

Use the original repository, not the fresh recovery clone.

Switch to September and restore the original data.

git switch --detach training-2026-09

dvc pull

python scripts/verify_dataset.py \
  --expected audit/2026-09.sha256 \
  --rows 270 \
  --as-of 2026-09-30

The initial verification should pass.

18.1. Deliberately modify the training file

Run:

python - <<'PY'
import csv
from pathlib import Path

path = Path("data/train.csv")

with path.open(
    "r",
    encoding="utf-8",
    newline="",
) as file:
    reader = csv.DictReader(file)
    fields = reader.fieldnames
    rows = list(reader)

if fields is None or not rows:
    raise SystemExit("Training file is empty")

rows[0]["unit_price_cents"] = str(
    int(rows[0]["unit_price_cents"]) + 500
)

with path.open(
    "w",
    encoding="utf-8",
    newline="",
) as file:
    writer = csv.DictWriter(
        file,
        fieldnames=fields,
        lineterminator="\n",
    )
    writer.writeheader()
    writer.writerows(rows)

print("Modified the first training record")
PY

This changes one feature value while leaving the training file’s row count unchanged.

Now run the verifier again.

python scripts/verify_dataset.py \
  --expected audit/2026-09.sha256 \
  --rows 270 \
  --as-of 2026-09-30

Expected failure:

Dataset verification failed: expected_digest, manifest_digest

The row count and snapshot date still agree with the historical manifest.

But the training file’s contents have changed.

This demonstrates why checking row counts alone is insufficient.

18.2. Restore the original artifact

Because we intentionally modified a DVC-managed workspace file, use a forced pull to restore the published historical version.

dvc pull --force

python scripts/verify_dataset.py \
  --expected audit/2026-09.sha256 \
  --rows 270 \
  --as-of 2026-09-30

The force option is appropriate in this controlled exercise because we deliberately want to discard the modified working file.

Do not use it indiscriminately in a workspace containing valuable uncommitted changes.

Expected output:

Dataset integrity verified
SHA-256: 9eb30129b4755d425927c0894362c7178d1b33e1a4397fcf0eb3362519f3c2d3
Rows: 270
As-of: 2026-09-30

Our historical artifact has been restored successfully.

18.3. Verify the transformation once more

Finally, force deterministic regeneration and verify the output independently.

dvc repro --force --no-run-cache

python scripts/verify_dataset.py \
  --expected audit/2026-09.sha256 \
  --rows 270 \
  --as-of 2026-09-30

This demonstrates three separate properties:

• The stored historical artifact can be recovered.

• Unauthorized local changes can be detected through integrity verification.

• The preserved inputs and transformation can regenerate the same published bytes.

Together, these are stronger guarantees than simply knowing that a pipeline still executes successfully.

19. Final principles to carry into production

The most important change in mindset is to stop treating a production training set as a query that can always be rerun.

A production training set is a historical artifact.

It represents a specific collection of records, feature values, labels, temporal rules, and transformation decisions.

That artifact deserves the same release discipline applied to production software.

The essential principles are:

• Preserve training datasets as identifiable, retained artifacts.

• Track upstream snapshots separately from transformation code.

• Record exact feature values, labels, and split membership.

• Use immutable references instead of mutable names.

• Verify artifact contents using cryptographic fingerprints.

• Preserve the runtime context required for regeneration.

• Coordinate data retention with model retention.

• Test historical restoration regularly from clean environments.

• Treat recoverability and integrity verification as separate guarantees.

Our DVC demonstration preserved two historical retail training datasets even though the same working filenames were reused.

We recovered September’s training file after October introduced new orders and historical corrections.

We verified that the recovered file matched its originally published SHA-256 digest.

We then regenerated the historical training set, checked the digest again, tested recovery from an independent clone, and detected a deliberate integrity violation.

The decisive question was never whether the current pipeline still ran.

It was whether we could identify, retrieve, and independently verify the exact historical dataset consumed by a particular model training run.

That is the foundation of trustworthy ML data lineage.

Your next action

Choose one model currently running in production.

Find its last training run and attempt to recover the exact dataset it consumed, including feature values, labels, and train/validation/test membership.

Verify that dataset against a retained digest or another trustworthy historical identity.

If you cannot complete that exercise today, introduce immutable dataset references, persisted training manifests, and an automated restoration test into your next training pipeline release.

Make historical dataset recovery a release requirement—not an emergency procedure.

Next month’s incident investigation should never depend on guessing what last month’s training data looked like.