Feature stores and reproducible training pipelines

A model is not reproducible merely because its training script has a fixed random seed. You must also reproduce the exact feature definitions, historical values, timestamps, dependency graph, and model inputs.

A feature store helps by making features versioned, discoverable data products rather than anonymous columns assembled inside a notebook. The registry records definitions, the offline path builds point-in-time-correct training sets, and the online path serves recent values during inference.

This tutorial builds a minimal local pipeline using Feast 0.64.0, MLflow 3.14.0, SQLite, Parquet, scikit-learn, and uv. Those Feast and MLflow releases were published in June 2026, while uv provides a lockfile-backed environment that can be checked into source control.

What reproducible training actually requires

A reproducible run has at least four coordinates:

• Code: the training and feature-definition commit.

• Environment: exact Python and package versions.

• Data: an immutable source snapshot or content digest.

• Time: the event timestamps used to reconstruct historical features.

The last coordinate is easy to miss.

Suppose a customer churned on June 15. A naive join might attach their current account balance, including transactions from June 20. The model then learns from information that did not exist when the prediction would have been made.

A point-in-time join instead retrieves the latest feature value at or before each label timestamp. Feast’s historical retrieval API is designed for this pattern and accepts entity keys plus timestamps as the upper bounds for feature lookup.

The pipeline we will build

The project has one batch feature source, a Feast registry, a SQLite online store, and an MLflow tracking database.

feature-pipeline/
├── feature_repo/
│   ├── data/
│   ├── feature_definitions.py
│   └── feature_store.yaml
├── artifacts/
├── make_data.py
├── train.py
├── score.py
├── pyproject.toml
└── uv.lock

Create the environment and pin the major tools used by the pipeline:

mkdir feature-pipeline
cd feature-pipeline

uv init --python 3.12
uv add feast==0.64.0 mlflow==3.14.0 pandas pyarrow scikit-learn
uv lock

mkdir -p feature_repo/data artifacts

Commit both pyproject.toml and uv.lock. In CI, use uv sync --locked or uv run --locked; the command fails rather than silently updating a stale lockfile.

Generate deterministic event-time data

Create make_data.py. It generates daily customer features and labels whose timestamps occur 30 minutes after their corresponding feature observations.

from pathlib import Path

import numpy as np
import pandas as pd


OUTPUT_DIR = Path("feature_repo/data")
OUTPUT_DIR.mkdir(parents=True, exist_ok=True)

rng = np.random.default_rng(7)
now = pd.Timestamp.now(tz="UTC").floor("h")

feature_rows: list[dict] = []
label_rows: list[dict] = []

for customer_id in range(1000, 1060):
    for day in range(45):
        event_timestamp = now - pd.Timedelta(days=44 - day)
        sessions_7d = max(0, int(rng.poisson(8)))
        spend_30d = round(float(rng.gamma(shape=2.5, scale=25.0)), 2)
        days_since_last_order = int(rng.integers(0, 35))

        feature_rows.append(
            {
                "customer_id": customer_id,
                "event_timestamp": event_timestamp,
                "created_timestamp": event_timestamp + pd.Timedelta(minutes=5),
                "sessions_7d": sessions_7d,
                "spend_30d": spend_30d,
                "days_since_last_order": days_since_last_order,
            }
        )

        if day >= 20 and day % 4 == 0:
            logit = (
                -0.8
                - 0.08 * sessions_7d
                - 0.006 * spend_30d
                + 0.12 * days_since_last_order
            )
            churn_probability = 1.0 / (1.0 + np.exp(-logit))

            label_rows.append(
                {
                    "customer_id": customer_id,
                    "event_timestamp": event_timestamp
                    + pd.Timedelta(minutes=30),
                    "churned": int(rng.random() < churn_probability),
                }
            )

features = pd.DataFrame(feature_rows)
labels = pd.DataFrame(label_rows)

features.to_parquet(OUTPUT_DIR / "customer_features.parquet", index=False)
labels.to_parquet(OUTPUT_DIR / "labels.parquet", index=False)

print(f"Wrote {len(features)} feature rows")
print(f"Wrote {len(labels)} labels")

The random generator is deterministic, but the timestamps remain recent enough to materialize into the online store. In a real system, replace this generator with an immutable table version, partition snapshot, or object-store path.

Define the feature contract

Create feature_repo/feature_store.yaml:

project: churn_demo
registry: data/registry.db
provider: local

online_store:
  type: sqlite
  path: data/online_store.db

entity_key_serialization_version: 3

Feast’s local provider uses a local registry, a file-based offline path, and SQLite for online serving. Production deployments can replace those components without changing how callers request feature services.

Now create feature_repo/feature_definitions.py:

from datetime import timedelta

from feast import Entity, FeatureService, FeatureView, Field, FileSource
from feast.types import Float32, Int64


customer = Entity(
    name="customer",
    join_keys=["customer_id"],
)

customer_source = FileSource(
    name="customer_source",
    path="data/customer_features.parquet",
    timestamp_field="event_timestamp",
    created_timestamp_column="created_timestamp",
)

customer_features = FeatureView(
    name="customer_features",
    entities=[customer],
    ttl=timedelta(days=60),
    schema=[
        Field(name="sessions_7d", dtype=Int64),
        Field(name="spend_30d", dtype=Float32),
        Field(name="days_since_last_order", dtype=Int64),
    ],
    source=customer_source,
    online=True,
    tags={"owner": "retention-ml"},
)

churn_v1 = FeatureService(
    name="churn_v1",
    features=[customer_features],
)

A feature service is the model-facing contract. Training and serving can both request churn_v1 instead of maintaining separate column lists. Feast’s current SDK supports Entity, FileSource, FeatureView, typed Field definitions, and feature services in this form.

Build and record the training set

Create train.py:

from pathlib import Path

import mlflow
import mlflow.sklearn
import pandas as pd
from feast import FeatureStore
from mlflow.models import infer_signature
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import train_test_split


REPO = Path("feature_repo")
ARTIFACTS = Path("artifacts")
ARTIFACTS.mkdir(exist_ok=True)

FEATURE_COLUMNS = [
    "sessions_7d",
    "spend_30d",
    "days_since_last_order",
]

store = FeatureStore(repo_path=str(REPO))
feature_service = store.get_feature_service("churn_v1")
labels = pd.read_parquet(REPO / "data/labels.parquet")

training_set = store.get_historical_features(
    entity_df=labels,
    features=feature_service,
).to_df()

training_path = ARTIFACTS / "training_set.parquet"
training_set.to_parquet(training_path, index=False)

X = training_set[FEATURE_COLUMNS]
y = training_set["churned"]

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.25,
    random_state=42,
    stratify=y,
)

model = RandomForestClassifier(
    n_estimators=200,
    max_depth=8,
    min_samples_leaf=3,
    random_state=42,
    n_jobs=1,
)
model.fit(X_train, y_train)

probabilities = model.predict_proba(X_test)[:, 1]
auc = roc_auc_score(y_test, probabilities)

mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("churn-reproducibility")

dataset = mlflow.data.from_pandas(
    training_set,
    source=str(training_path),
    targets="churned",
    name="churn-training-set",
)

with mlflow.start_run() as run:
    mlflow.log_input(dataset, context="training")
    mlflow.log_params(model.get_params())
    mlflow.log_param("feature_service", "churn_v1")
    mlflow.log_metric("test_roc_auc", auc)

    mlflow.log_artifact("uv.lock", artifact_path="reproducibility")
    mlflow.log_artifact(
        REPO / "feature_store.yaml",
        artifact_path="reproducibility",
    )
    mlflow.log_artifact(
        REPO / "feature_definitions.py",
        artifact_path="reproducibility",
    )
    mlflow.log_artifact(
        REPO / "data/registry.db",
        artifact_path="reproducibility",
    )
    mlflow.log_artifact(training_path, artifact_path="datasets")

    signature = infer_signature(X_test, model.predict(X_test))
    model_info = mlflow.sklearn.log_model(
        sk_model=model,
        name="model",
        signature=signature,
        input_example=X_train.head(3),
    )

    (ARTIFACTS / "model_uri.txt").write_text(
        model_info.model_uri,
        encoding="utf-8",
    )

    print(f"Run ID: {run.info.run_id}")
    print(f"ROC AUC: {auc:.3f}")
    print(f"Model URI: {model_info.model_uri}")

MLflow datasets record source, schema, profile, target, and a digest. Model signatures provide an input/output contract, while input examples validate and document expected requests. The current model API uses name; artifact_path is deprecated.

Run data generation, register the feature definitions, and train:

uv run --locked python make_data.py
uv run --locked feast -c feature_repo apply
uv run --locked python train.py

Materialize and score online features

Historical retrieval reads the offline source. Online inference requires the latest values to be materialized into SQLite.

MATERIALIZE_TO="$(date -u +%Y-%m-%dT%H:%M:%S)"
uv run --locked feast -c feature_repo materialize-incremental \
  "$MATERIALIZE_TO"

Feast materialization loads recent feature values into the configured online store. get_online_features() then retrieves those values by entity key for low-latency inference.

Create score.py:

from pathlib import Path

import mlflow
import pandas as pd
from feast import FeatureStore


FEATURE_COLUMNS = [
    "sessions_7d",
    "spend_30d",
    "days_since_last_order",
]

mlflow.set_tracking_uri("sqlite:///mlflow.db")

store = FeatureStore(repo_path="feature_repo")
feature_service = store.get_feature_service("churn_v1")

response = store.get_online_features(
    features=feature_service,
    entity_rows=[
        {"customer_id": 1001},
        {"customer_id": 1002},
        {"customer_id": 1003},
    ],
).to_dict()

online_frame = pd.DataFrame(response)
model_uri = Path("artifacts/model_uri.txt").read_text(
    encoding="utf-8"
).strip()

model = mlflow.pyfunc.load_model(model_uri)
predictions = model.predict(online_frame[FEATURE_COLUMNS])

result = online_frame[["customer_id", *FEATURE_COLUMNS]].copy()
result["predicted_churn"] = predictions

print(result.to_string(index=False))

Execute it with the same locked environment:

uv run --locked python score.py

Guardrails for production pipelines

The demonstration is local, but the reproducibility rules scale:

• Treat feature definitions and feature services as reviewed application code.

• Version the registry or export a registry snapshot with every model run.

• Store immutable dataset references, not mutable table names such as latest.

• Log label-query parameters, feature-service names, time ranges, and time zones.

• Fail training when required features are null, stale, duplicated, or outside valid ranges.

• Run offline-versus-online parity tests for representative entities.

• Promote a model only when its feature service exists in the target environment.

• Make backfills explicit; never let a routine retry silently rewrite historical partitions.

• Monitor feature freshness separately from model latency and model accuracy.

A feature store reduces duplicated logic, but it does not automatically make upstream SQL, streaming aggregations, labels, or orchestration reproducible.

Cherry on the cake: the anti-skew tool had a skew-shaped bug

In May 2026, Feast merged a fix for a local compute-engine bug involving renamed entity join keys. A source could map USERID to user_id, after which downstream deduplication and join nodes still searched for the original name. Depending on the path, the result could be a KeyError or an empty join.

The fix shipped in Feast 0.64.0.

The surprising lesson is that adopting a feature store is not the end of train/serve-skew prevention. You must also pin the feature-store version, test renamed columns and entity mappings, and compare offline and online vectors before promotion.

Start by implementing this pipeline locally, commit its lockfile and registry snapshot, and then add one CI test that retrieves the same entity through both the historical and online paths. That single test can catch an entire class of expensive production failures before your next model ships.