Orchestration beyond cron: DAGs, retries, and backfills with Airflow or Dagster

LEARN · MODERN DATA ENGINEERING

Cron tells you when. An orchestrator tells you what happened.

Cron is excellent at one job: starting a command at a particular time.

That simplicity becomes a liability once a workflow has dependencies, partial failures, historical reprocessing, concurrency limits, or data that belongs to a logical time period rather than the wall-clock moment when a process happens to start.

Consider a daily retail pipeline:

Extract the previous day’s orders.
Validate that the extract is non-empty and structurally sane.
Aggregate revenue by store.
Publish the result.
Refresh a dashboard only after publishing succeeds.

A crontab can launch five scripts. It cannot naturally answer the questions operators start asking after the first serious failure:

Did aggregation run even though extraction failed?
Which step should be retried?
Has a failed step already produced side effects?
What date of data was this execution supposed to process?
Can we safely recompute the previous 90 days?
What happens if tomorrow’s run starts while today’s run is still executing?
If a worker dies after committing output but before reporting success, should that work execute again?

Those questions are the territory of workflow orchestration.

A useful mental model is:

cron:
time -> command

orchestrator:
trigger -> run -> dependency graph -> task state -> retries -> outputs -> history

The important difference is not the graph visualization. It is persisted execution state.

An orchestrator knows that a particular run exists, which units of work belong to it, which ones succeeded, which ones failed, which ones are waiting on dependencies, and which historical interval the run represents.

As of August 17, 2026, Apache Airflow 3.3.1 is the current stable Airflow release; it was released on August 12, 2026. Airflow 3.x exposes its supported DAG-authoring interface through airflow.sdk. Dagster’s current documentation identifies version 1.13.18 as its latest release.

Those are the versions used throughout this lesson.

A DAG is more than a prettier crontab

A directed acyclic graph, or DAG, represents dependencies between units of work.

A simple retail pipeline might look like this:

extract_orders
      |
      v
validate_orders
      |
      v
aggregate_daily_sales
      |
      v
publish_metrics
      |
      v
refresh_dashboard

The arrows encode operational meaning.

If validate_orders fails, the scheduler knows that aggregate_daily_sales should not start. If publishing succeeds but the dashboard refresh fails, the orchestrator can retry the refresh rather than blindly starting everything again.

A shell script can approximate dependency ordering:

python extract.py &&
python validate.py &&
python aggregate.py &&
python publish.py &&
python refresh.py

That is useful, but it does not give you a durable scheduler-level model of runs, retries, historical intervals, task attempts, concurrency, or backfills.

Once those features matter, teams that remain on cron usually start implementing a scheduler around their scripts.

That is the point at which using an actual orchestrator becomes simpler than continuing to extend cron.

The most important object is the run

An orchestrated execution should be thought of as a first-class run rather than “the process that happened to start at midnight.”

That distinction matters because there are several different clocks in a production pipeline:

When the scheduler created the run.
When a worker started executing it.
When the source data was generated.
The logical interval the run is responsible for.
When downstream data became visible.

Those timestamps are not interchangeable.

Suppose a daily pipeline for August 10 fails and is retried on August 11.

The second attempt still needs to compute August 10.

Likewise, if you backfill August 1 through August 10 on August 17, those executions must process their assigned historical intervals rather than whatever happens to be “today.”

This is why logical time is foundational to retries and backfills.

Retries are safe only when the work is replayable

Retries sound straightforward:

If the task fails, try it again.

The problem is that “failed” does not necessarily mean “nothing happened.”

Consider this function:

def publish_daily_revenue(connection, day, revenue):
    connection.execute(
        """
        INSERT INTO daily_revenue(day, revenue)
        VALUES (?, ?)
        """,
        (day, revenue),
    )
    connection.commit()

    notify_remote_service(day)

Imagine that the database commit succeeds.

Then the network call fails.

The task reports failure, so the orchestrator retries it.

The second attempt performs the insert again.

If the destination permits duplicates, you now have duplicate business state even though the retry mechanism behaved exactly as configured.

Reliable retries therefore depend on three properties.

1. Deterministic input boundaries

A task should know exactly which partition or interval it is processing.

Bad pipeline logic asks questions like:

What is today’s date?
Which records arrived during the last 24 hours relative to right now?
What does the source currently contain?

Replayable logic asks:

Which partition did the orchestrator assign this run?
What are the explicit start and end boundaries?
Which immutable or versioned inputs belong to that interval?

A retry five minutes later must not silently process a different dataset from the first attempt.

2. Idempotent side effects

Repeated execution should converge to the same externally visible state as one successful execution.

Common techniques include:

Upserting using a stable business key.
Replacing one complete partition atomically.
Deleting and rebuilding one partition inside a transaction.
Writing immutable output to a deterministic object-storage key.
Using an idempotency key when an external API supports one.
Recording processed event identifiers under a unique constraint.

Idempotency does not necessarily mean that no code runs twice.

It means running twice does not create two business effects.

3. Bounded retries

Retries should also have limits.

A sensible retry policy normally considers:

Maximum attempt count.
Delay between attempts.
Exponential backoff.
Jitter when many workers could retry together.
Whether an error is transient or permanent.

Airflow 3.3.1 supports numeric values for retry_exponential_backoff; for example, 2.0 specifies a doubling multiplier between retry delays.

A malformed payload should generally fail quickly.

A temporary connection reset may deserve a retry.

Those are different operational conditions and should not be treated identically.

Build a runnable Airflow 3.3.1 retry experiment

The fastest way to understand these semantics is to deliberately create the ambiguous failure we just discussed.

We will build a small daily workflow that:

Generates deterministic synthetic retail metrics.
Uses the run’s data interval as the partition key.
Writes that partition to SQLite.
Commits successfully.
Intentionally crashes after the first commit.
Retries automatically.
Upserts the same partition instead of duplicating it.
Verifies that exactly one destination row exists.
Can later be backfilled over historical dates using unchanged DAG code.

Airflow 3.3.1 is tested with Python 3.10 through Python 3.14. For local development, Airflow documents SQLite as acceptable but explicitly recommends a production-grade database for real deployments.

Create an isolated environment:

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

export AIRFLOW_HOME="$HOME/airflow"
export AIRFLOW_VERSION="3.3.1"
export PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
export CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

python -m pip install \
  "apache-airflow==${AIRFLOW_VERSION}" \
  --constraint "${CONSTRAINT_URL}"

mkdir -p "$AIRFLOW_HOME/dags"

That constraints URL is intentionally a raw URL rather than Markdown link syntax. Airflow’s current installation guidance recommends release-specific constraint files when installing through pip.

Now save the following as $AIRFLOW_HOME/dags/orchestration_demo.py:

from __future__ import annotations

import os
import random
import sqlite3
from datetime import timedelta
from pathlib import Path

import pendulum

from airflow.sdk import dag, get_current_context, task


def database_path() -> Path:
    airflow_home = Path(
        os.environ.get("AIRFLOW_HOME", str(Path.home() / "airflow"))
    )
    airflow_home.mkdir(parents=True, exist_ok=True)
    return airflow_home / "demo_orders.sqlite"


def metrics_for_day(partition_date: str) -> tuple[int, float]:
    seed = int(partition_date.replace("-", ""))
    rng = random.Random(seed)

    order_count = rng.randint(80, 180)
    order_values = [
        round(rng.uniform(8.0, 140.0), 2)
        for _ in range(order_count)
    ]

    revenue = round(sum(order_values), 2)
    return order_count, revenue


@dag(
    dag_id="orchestration_demo",
    schedule="@daily",
    start_date=pendulum.datetime(2026, 8, 1, tz="UTC"),
    catchup=False,
    tags=["course-demo"],
)
def orchestration_demo():
    @task
    def build_daily_metrics() -> dict[str, object]:
        context = get_current_context()
        data_interval_start = context["data_interval_start"]

        partition_date = data_interval_start.date().isoformat()
        order_count, revenue = metrics_for_day(partition_date)

        return {
            "partition_date": partition_date,
            "order_count": order_count,
            "revenue": revenue,
        }

    @task(
        retries=2,
        retry_delay=timedelta(seconds=5),
        retry_exponential_backoff=2.0,
    )
    def publish_metrics(
        metrics: dict[str, object],
    ) -> dict[str, object]:
        context = get_current_context()
        task_instance = context["ti"]

        partition_date = str(metrics["partition_date"])
        order_count = int(metrics["order_count"])
        revenue = float(metrics["revenue"])

        db_path = database_path()

        with sqlite3.connect(db_path) as connection:
            connection.execute(
                """
                CREATE TABLE IF NOT EXISTS daily_order_metrics (
                    partition_date TEXT PRIMARY KEY,
                    order_count INTEGER NOT NULL,
                    revenue REAL NOT NULL
                )
                """
            )

            connection.execute(
                """
                INSERT INTO daily_order_metrics (
                    partition_date,
                    order_count,
                    revenue
                )
                VALUES (?, ?, ?)
                ON CONFLICT(partition_date) DO UPDATE SET
                    order_count = excluded.order_count,
                    revenue = excluded.revenue
                """,
                (partition_date, order_count, revenue),
            )

            connection.commit()

        print(
            f"Committed partition={partition_date}, "
            f"orders={order_count}, "
            f"revenue={revenue}, "
            f"try={task_instance.try_number}"
        )

        if task_instance.try_number == 1:
            raise RuntimeError(
                "Intentional demo failure after the database commit"
            )

        return metrics

    @task
    def verify_partition(
        metrics: dict[str, object],
    ) -> None:
        partition_date = str(metrics["partition_date"])
        expected_order_count = int(metrics["order_count"])
        expected_revenue = float(metrics["revenue"])

        with sqlite3.connect(database_path()) as connection:
            rows = connection.execute(
                """
                SELECT partition_date, order_count, revenue
                FROM daily_order_metrics
                WHERE partition_date = ?
                """,
                (partition_date,),
            ).fetchall()

        assert len(rows) == 1, (
            f"Expected exactly one row for {partition_date}, "
            f"found {len(rows)}"
        )

        _, order_count, revenue = rows[0]

        assert order_count == expected_order_count
        assert revenue == expected_revenue

        print(
            f"Verified exactly one correct row for {partition_date}"
        )

    metrics = build_daily_metrics()
    published = publish_metrics(metrics)
    verify_partition(published)


orchestration_demo()

There are three important things happening here.

First, DAG authoring uses the supported airflow.sdk namespace, which Airflow 3 documents as the primary public interface for DAG authors. get_current_context() is also part of that interface.

Second, the workflow uses data_interval_start to identify the business partition. For a scheduled daily DAG, Airflow associates each DAG run with a data interval; its logical date denotes the start of that interval rather than the wall-clock time at which execution begins.

Third, the failure occurs at exactly the uncomfortable point:

database transaction committed
          |
          v
worker raises exception
          |
          v
orchestrator sees failure
          |
          v
task retry executes

The database already contains the result before Airflow sees the intentional exception.

The retry therefore repeats the write.

Because the table has a primary key on partition_date and the statement uses ON CONFLICT ... DO UPDATE, the repeated write converges on one correct row instead of creating duplicates.

That is the behavior we want from production retries.

Run the Airflow experiment

Start Airflow locally:

airflow standalone

airflow standalone is a current Airflow 3.3.1 command intended for local development and testing. It starts an all-in-one Airflow installation; it is not the recommended production deployment model.

Open the Airflow UI, locate orchestration_demo, unpause it if necessary, and trigger a run.

Watch publish_metrics.

Its first attempt should:

Write the partition.
Commit the transaction.
Raise the intentional exception.
Enter retry handling.

The next attempt writes exactly the same partition and succeeds.

verify_partition then checks that the destination contains one row rather than two.

You can inspect the database separately with:

python - <<'PY'
import os
import sqlite3
from pathlib import Path

airflow_home = Path(
    os.environ.get("AIRFLOW_HOME", str(Path.home() / "airflow"))
)
db_path = airflow_home / "demo_orders.sqlite"

with sqlite3.connect(db_path) as connection:
    rows = connection.execute(
        """
        SELECT partition_date, order_count, revenue
        FROM daily_order_metrics
        ORDER BY partition_date
        """
    ).fetchall()

for row in rows:
    print(row)
PY

After multiple partitions have run, the shape should look like this:

('2026-08-10', 133, 10089.24)
('2026-08-11', 118, 8284.71)
('2026-08-12', 154, 11412.58)

The exact generated values depend deterministically on the date.

What matters is that each partition date appears once.

The destination should not look like this:

A scheduler can repeat execution.

It is your storage and side-effect design that determines whether repeating execution is safe.

Retry state is not business state

This leads to one of the most important architecture rules in orchestration:

The orchestrator’s metadata should tell you whether work ran. The destination must enforce whether business state is valid.

Do not use “Airflow says this task succeeded once” as your only duplicate-prevention mechanism.

A worker can:

Commit a database transaction.
Lose its network connection.
Crash before reporting success.
Be considered failed by orchestration infrastructure.
Execute again.

No scheduler can make that boundary disappear.

This is a distributed-systems ambiguity: the remote side effect may have committed even when the caller does not know whether it committed.

Your destination therefore needs its own correctness controls, such as:

Primary keys.
Unique constraints.
Upserts.
Partition replacement.
Idempotency keys.
Transactional outboxes.
Compare-and-swap semantics.
Immutable versioned object keys.

Scheduler state is operational metadata.

It is not a replacement for business invariants.

Backfills solve the problem cron never really modeled

Eventually someone will discover that a transformation was incorrect for a historical range.

Suppose revenue was computed incorrectly from August 3 through August 11.

The fix is not to execute “today’s job” nine times.

You need nine historical executions, each assigned to the appropriate time interval.

That is a backfill.

Airflow 3.3.1 provides current backfill support through the UI, CLI, and REST API. For CLI-driven backfills, the command is airflow backfill create; old Airflow 2-era tutorials using airflow dags backfill should not be copied into new Airflow 3 projects.

For the demonstration DAG:

airflow backfill create \
  --dag-id orchestration_demo \
  --from-date 2026-08-03 \
  --to-date 2026-08-11 \
  --reprocess-behavior completed \
  --max-active-runs 1

Airflow 3.3.1 defines three reprocessing behaviors.

none: if a run already exists for that logical date, do not create another.
failed: create another run when the existing run failed.
completed: permit another run when the latest existing run is completed or failed.

If the latest run for a logical date is still queued or running, Airflow does not create another run for that date regardless of the reprocessing mode.

The --max-active-runs 1 setting is deliberately conservative for this local SQLite example.

Airflow’s backfill-specific maximum-active-runs control operates independently from the DAG’s ordinary max_active_runs setting.

That distinction matters in production because historical reprocessing creates a very different load profile from ordinary scheduling.

A 365-day backfill is 365 production workloads

Imagine the normal daily pipeline performs:

1 source extraction
1 warehouse transformation
1 publication step

A 365-day backfill may perform:

365 source extractions
365 warehouse transformations
365 publication operations

If you run 50 historical partitions simultaneously, you are effectively asking downstream infrastructure to tolerate something like 50 days of scheduled work arriving together.

That can overwhelm:

Source databases.
REST APIs.
Warehouse compute pools.
Lock managers.
Object stores.
Message brokers.
Downstream consumers.
External notification systems.

It can also overwhelm your budget.

Backfill concurrency should therefore be an explicit capacity decision rather than whatever value finishes fastest.

Before increasing it, inspect:

Source-system rate limits.
Database connection limits.
Warehouse concurrency.
Partition size.
Lock contention.
API quotas.
Downstream publication behavior.
Cost per partition.
Normal production workload competing for the same resources.

Airflow also exposes --run-backwards as a backfill CLI control when you want newer logical intervals to execute before older ones. It is specifically documented as a CLI option rather than an API flag.

For example:

airflow backfill create \
  --dag-id orchestration_demo \
  --from-date 2026-08-03 \
  --to-date 2026-08-11 \
  --reprocess-behavior completed \
  --max-active-runs 2 \
  --run-backwards

Newest-first processing can be useful when recent repaired data has higher business priority.

Oldest-first processing can be useful when downstream intervals depend on previous state.

Neither ordering is universally correct.

Catchup and backfill are related, but different

These concepts are often confused.

Catchup asks whether the scheduler should automatically create missed scheduled intervals.

Backfill is an explicit request to create historical runs over a selected range.

Airflow’s current default configuration has scheduler catchup disabled unless it is turned on. A DAG can also declare this explicitly with catchup=False.

Our DAG does exactly that:

@dag(
    dag_id="orchestration_demo",
    schedule="@daily",
    start_date=pendulum.datetime(2026, 8, 1, tz="UTC"),
    catchup=False,
)
def orchestration_demo():
    ...

This means deployment does not automatically generate every missing interval merely because start_date is historical.

It does not mean operators cannot intentionally backfill history.

That is often a strong production posture:

Routine deployment does not accidentally launch months of historical work.
Historical processing requires an explicit action.
Backfill concurrency can be controlled separately.
Reprocessing behavior is chosen deliberately.

Automatic history and operator-requested history are different policies.

Logical time beats wall-clock time

A backfillable pipeline must not derive its business partition from datetime.now().

Consider:

from datetime import datetime, timezone


def current_partition() -> str:
    return datetime.now(timezone.utc).date().isoformat()

Now suppose you execute an August 4 backfill on August 17.

The orchestrator believes it is processing August 4.

Your function writes August 17.

The scheduler cannot rescue business logic that ignores the scheduler’s interval.

For the scheduled Airflow DAG in this lesson, the correct pattern is:

from airflow.sdk import get_current_context


def assigned_partition() -> str:
    context = get_current_context()
    return context["data_interval_start"].date().isoformat()

Airflow’s current documentation describes each scheduled DAG run as having a data interval and distinguishes that interval from wall-clock execution time. It also warns that manually triggered runs can have timetable-dependent interval semantics, so code should deliberately choose whether it needs the data interval or an explicitly supplied logical date.

The larger design rule is portable across orchestrators:

Time should be explicit input, not ambient global state.

Conceptually, historical computation should resemble:

output_for_partition =
    transform(input_for_partition, code_version)

not:

output =
    transform(whatever_the_world_looks_like_right_now)

That difference is what makes replay meaningful.

Idempotency means convergence, not merely suppressing errors

A common anti-pattern is to interpret “ignore duplicate errors” as idempotency.

Consider:

def save_order(connection, order_id, amount):
    try:
        connection.execute(
            """
            INSERT INTO orders(order_id, amount)
            VALUES (?, ?)
            """,
            (order_id, amount),
        )
    except Exception:
        pass

This is dangerous for several reasons.

The broad exception handler hides unrelated failures.

More importantly, suppose an earlier pipeline bug wrote:

order_id=123
amount=40.00

You later correct the transformation and discover that the historical value should have been:

order_id=123
amount=44.00

A backfill reaches the existing row, suppresses the duplicate error, and leaves the incorrect value untouched.

The operation is “quiet.”

It is not correct.

An upsert better expresses convergence:

connection.execute(
    """
    INSERT INTO orders(order_id, amount)
    VALUES (?, ?)
    ON CONFLICT(order_id) DO UPDATE SET
        amount = excluded.amount
    """,
    (order_id, amount),
)

Now repeated execution moves the destination toward the desired state.

For a full daily aggregate, you might instead replace the complete partition.

For an immutable event log, the right behavior might be to reject a repeated event ID while preserving the original row.

For an external payment or messaging API, you might need an operation-specific idempotency key.

Idempotency is a business property, not a particular SQL keyword.

Side effects deserve stricter treatment than data transforms

Backfills become especially dangerous when historical computation also produces irreversible external actions.

Suppose your DAG does this:

recompute_order_status
        |
        v
write_warehouse_table
        |
        v
send_customer_email

Recomputing a table partition may be perfectly safe.

Re-sending 90 days of customer emails may be catastrophic.

The usual solution is to separate historical state reconstruction from live side effects.

For example:

scheduled live run
    |
    +--> rebuild state
    |
    +--> emit allowed live notification

historical backfill
    |
    +--> rebuild state
    |
    +--> suppress or separately reconcile notification

Do not assume that because a data write is idempotent, every downstream operation is also idempotent.

Audit side effects individually:

Emails.
Push notifications.
Payment requests.
Refunds.
CRM updates.
Support tickets.
Webhooks.
Inventory reservations.
Search indexing.
Cache invalidations.

A workflow can be mathematically replayable while still producing disastrous external behavior.

Dagster 1.13.18 starts from a different mental model

Airflow naturally encourages you to think in terms of workflows and tasks.

Dagster often encourages you to begin with the data assets themselves:

raw_orders
    |
    v
daily_order_metrics
    |
    v
executive_metrics

Dagster 1.13.18 is the current documented release. Its current guidance strongly favors assets for new data pipelines while retaining lower-level ops for procedural cases.

An asset can declare:

Its partition definition.
Its dependencies.
Its retry policy.
Its backfill policy.

That makes historical processing feel less like “rerun this job 100 times” and more like “materialize these 100 missing or corrected partitions.”

Here is a current Dagster example using the same synthetic retail idea:

from __future__ import annotations

import random
import sqlite3
from pathlib import Path

import dagster as dg


DAILY_PARTITIONS = dg.DailyPartitionsDefinition(
    start_date="2026-08-01"
)

DATABASE_PATH = Path("dagster_orders.sqlite")


def metrics_for_day(partition_date: str) -> tuple[int, float]:
    seed = int(partition_date.replace("-", ""))
    rng = random.Random(seed)

    order_count = rng.randint(80, 180)
    order_values = [
        round(rng.uniform(8.0, 140.0), 2)
        for _ in range(order_count)
    ]

    revenue = round(sum(order_values), 2)
    return order_count, revenue


@dg.asset(
    partitions_def=DAILY_PARTITIONS,
    backfill_policy=dg.BackfillPolicy.multi_run(
        max_partitions_per_run=5
    ),
    retry_policy=dg.RetryPolicy(
        max_retries=2,
        delay=2,
        backoff=dg.Backoff.EXPONENTIAL,
        jitter=dg.Jitter.PLUS_MINUS,
    ),
)
def daily_order_metrics(
    context: dg.AssetExecutionContext,
) -> None:
    for partition_date in context.partition_keys:
        order_count, revenue = metrics_for_day(partition_date)

        with sqlite3.connect(DATABASE_PATH) as connection:
            connection.execute(
                """
                CREATE TABLE IF NOT EXISTS daily_order_metrics (
                    partition_date TEXT PRIMARY KEY,
                    order_count INTEGER NOT NULL,
                    revenue REAL NOT NULL
                )
                """
            )

            connection.execute(
                """
                INSERT INTO daily_order_metrics (
                    partition_date,
                    order_count,
                    revenue
                )
                VALUES (?, ?, ?)
                ON CONFLICT(partition_date) DO UPDATE SET
                    order_count = excluded.order_count,
                    revenue = excluded.revenue
                """,
                (partition_date, order_count, revenue),
            )

            connection.commit()

        context.log.info(
            f"Materialized {partition_date}: "
            f"{order_count} orders, revenue={revenue}"
        )


defs = dg.Definitions(
    assets=[daily_order_metrics]
)

The APIs in that example are current in Dagster 1.13.18.

Dagster’s current documentation recommends constructing policies with BackfillPolicy.multi_run(...) or BackfillPolicy.single_run() instead of constructing BackfillPolicy directly.

context.partition_keys is also the current property for code that needs to process multiple selected partitions in one run. Older asset_partition_keys_for_output APIs are deprecated in favor of properties such as partition_keys.

One partition per run, batches, or one giant run?

Partition granularity determines failure granularity.

Suppose you need to backfill 100 daily partitions.

One run per partition

100 partitions
      |
      v
100 runs

Advantages:

Excellent failure isolation.
One bad partition does not invalidate the other 99.
Cheap retries.
Clear per-partition observability.

Costs:

More scheduler overhead.
More worker startup overhead.
More infrastructure churn for very short tasks.

Batched partitions

With batches of 10:

100 partitions
      |
      v
10 runs
      |
      v
10 partitions per run

Advantages:

Lower scheduler and startup overhead.
Still provides reasonable recovery granularity.
Useful for short partition computations.

Costs:

One failing partition can cause its batch to be retried.
Your implementation must actually handle multiple partition keys.

That is why the Dagster example uses:

for partition_date in context.partition_keys:
    order_count, revenue = metrics_for_day(partition_date)

It does not assume the run contains exactly one partition.

Dagster’s documented BackfillPolicy.multi_run(max_partitions_per_run=10) behavior groups a 100-partition backfill into runs containing at most 10 partitions.

Single-run backfill

You can also process the entire range in one run:

100 partitions
      |
      v
1 run

Dagster exposes that model through:

backfill_policy = dg.BackfillPolicy.single_run()

The current backfill documentation recommends range-aware context properties such as partition_time_window, partition_key_range, or partition_keys when implementing single-run historical processing.

A single run can be excellent when the underlying compute engine naturally operates on ranges.

For example, a warehouse might efficiently execute one statement that atomically rewrites 100 partitions.

The trade-off is blast radius.

If the giant run fails after 95 percent of its work, recovery may be expensive unless the underlying operation itself has checkpoints or atomic semantics.

A good heuristic is:

The right unit of orchestration should usually resemble the right unit of recovery.

Step retries and whole-run retries are different tools

Dagster makes another useful distinction: retrying a failed computation step is not the same as retrying a failed run.

A computation-level retry can be defined with:

retry_policy = dg.RetryPolicy(
    max_retries=3,
    delay=2,
    backoff=dg.Backoff.EXPONENTIAL,
    jitter=dg.Jitter.PLUS_MINUS,
)

Dagster’s current RetryPolicy supports a maximum retry count, base delay, backoff modifier, and jitter modifier.

That is appropriate for a step that raises an ordinary transient exception.

Whole-run retries solve another class of problem: the process executing the run may itself crash or be unexpectedly terminated.

For Dagster Open Source, current run-retry configuration lives in dagster.yaml:

run_retries:
  enabled: true
  max_retries: 2
  retry_on_asset_or_op_failure: false

With retry_on_asset_or_op_failure: false, whole-run retries are reserved for failures such as unexpected run-process termination rather than ordinary asset or op exceptions.

This matters because retry layers multiply.

Suppose:

step retries = 3
whole-run retries = 3

A run-level retry can reset the step’s retry counter.

The result may be substantially more executions than an engineer expects from casually reading either setting by itself. Dagster’s current documentation explicitly warns about this interaction.

A strong default principle is:

Retry a flaky API call at the smallest sensible computation boundary.
Retry the run when infrastructure destroyed the run itself.
Do not stack retry mechanisms without calculating the maximum resulting attempts.

The cherry on the cake: a backfill that doubled revenue

Non-idempotent backfills are not merely a theoretical warning.

A July 28, 2026 first-person engineering write-up documents a real incident in which a team discovered a month of corrupted data, applied a small code fix, and reran roughly 30 days of historical processing. The backfill wrote using append-style INSERT behavior rather than replacing existing partitions. The result was that historical rows were stacked on top of the original rows, causing downstream dashboards to show roughly doubled revenue. The author says the team then spent a full day deleting duplicate partitions, rerunning correctly, and explaining why the affected historical numbers were invalid. The author explicitly identifies the opening incident as a real story.

The bug that initiated the recovery was small.

The recovery operation became a second incident because the backfill was non-idempotent.

The sequence is worth remembering:

historical data is wrong
        |
        v
engineer fixes transformation
        |
        v
backfill reprocesses existing partitions
        |
        v
destination blindly appends
        |
        v
historical rows now exist twice
        |
        v
aggregates double

That is exactly why “the code ran successfully” is a weak production correctness signal.

Every backfill run in that scenario could have been perfectly green.

The business output was still wrong.

The orchestration system can faithfully automate whatever semantics you give it, including destructive ones.

Backfills should be boring

A well-designed backfill should be operationally uninteresting.

You should be able to say:

Run the production transformation for partitions A through B.

You should not need to create:

backfill_final.py
backfill_final_v2.py
backfill_final_v2_fixed.py
backfill_final_v2_fixed_really.py

One-off historical scripts tend to:

Drift from normal production logic.
Receive weaker test coverage.
Bypass dependency rules.
Bypass normal observability.
Accumulate ad hoc cleanup behavior.
Be written during incidents when engineers are under pressure.
Disappear afterward without becoming reusable infrastructure.

The safer model is to keep the transformation path constant and vary only the assigned partition or interval.

That usually implies:

Time boundaries are explicit.
Raw or reconstructable inputs are retained.
Outputs are partition-addressable.
Writes are repeatable.
Side effects are separately controlled.
Historical executions are observable in the same system as scheduled executions.
Code versions are traceable.
Validation happens after reconstruction.

Backfill should be a normal execution mode, not a custom emergency program.

Validation must check data, not just scheduler state

A green orchestrator means execution completed according to the orchestrator’s model.

It does not prove that the result is correct.

For historical processing, useful validation can include:

Expected row count ranges.
Unique-key checks.
Partition completeness.
Sum or count reconciliation against source records.
Comparison against an independently computed aggregate.
Schema validation.
Null-rate limits.
Duplicate detection.
Checksums for deterministic exports.

For the simple SQLite tutorial, our assertion is intentionally small:

assert len(rows) == 1

In a production revenue pipeline, you might also verify that the recomputed aggregate equals an independently calculated source total.

The operating pattern should be:

compute
  |
  v
write
  |
  v
validate
  |
  +--> publish if valid
  |
  +--> stop if invalid

Not:

compute
  |
  v
write
  |
  v
task turned green
  |
  v
assume everything is correct

The more powerful your backfill machinery becomes, the more important these validation gates become.

Plan for backfill amplification

Backfills amplify more than compute.

Consider a pipeline that normally performs one of each operation per day:

One source query.
One transformation.
One search-index update.
One webhook.
One cache invalidation.

Now backfill six months.

You may have requested historical data recomputation.

You may accidentally have requested:

180 source queries.
180 index rebuilds.
180 webhook batches.
180 cache invalidations.
A sudden burst of downstream event traffic.

This is why serious backfill planning should answer questions such as:

Which date range is being rebuilt?
Which exact assets or tables are affected?
Which code version is being used?
Are source records still immutable?
Which downstream systems will receive new events?
Are notifications suppressed?
What is the maximum parallelism?
How will normal scheduled runs coexist with the backfill?
What happens if the backfill is cancelled halfway through?
Can it safely resume?
What validation determines success?
Can repaired data be staged before becoming visible?

Treat a large backfill like a production change.

Because it is one.

Design retries around the failure you actually expect

“Retries enabled” is not a reliability strategy.

A better approach is to classify failures.

Transient infrastructure failure

Examples:

Connection reset.
Temporary DNS failure.
HTTP 503.
Short database failover.

Potential response:

Retry.
Exponential backoff.
Jitter.
Limited attempts.

Rate limiting

Examples:

HTTP 429.
Warehouse concurrency exhaustion.
SaaS quota reached.

Potential response:

Respect server-provided retry timing where available.
Increase delay.
Reduce concurrency.
Avoid synchronized workers.

Permanent configuration failure

Examples:

Invalid credentials.
Missing table.
Malformed configuration.
Forbidden API permission.

Potential response:

Fail quickly.
Alert.
Do not hammer the dependency repeatedly.

Bad business input

Examples:

Impossible order value.
Unknown currency code.
Broken schema.
Required field missing.

Potential response:

Quarantine or explicitly fail.
Do not treat malformed data as a transient outage.

A retry mechanism is only useful when another attempt has a realistic chance of producing a different outcome.

Airflow or Dagster?

Both can orchestrate serious production pipelines.

The useful question is not which tool “wins.” It is which abstraction makes correctness easiest for your system.

Airflow is natural when the workflow is the primary object

If your architecture is easiest to explain as:

extract
   |
   v
transform
   |
   v
publish

Airflow’s DAG model is intuitive.

It provides:

Task dependencies.
Scheduled DAG runs.
Task retries.
Run history.
Backfill controls.
Concurrency management.
A broad provider ecosystem.
A stable authoring interface through airflow.sdk in Airflow 3.

Dagster is natural when data assets are the primary object

If operators think in terms of:

raw_orders
    |
daily_sales
    |
revenue_report

Dagster’s asset-centric model can be especially compelling.

Partitions, materializations, asset lineage, retry policies, and asset backfills live close to the data concepts themselves. Dagster’s current documentation recommends assets over ops when building new data pipelines.

Practical questions matter more than philosophy:

Do operators primarily reason about jobs or datasets?
Are partitioned historical workloads central to the system?
How important is asset lineage during debugging?
Which integrations does the organization already depend on?
Which tool does the platform team already know how to operate?
How will compute workers be launched?
How are credentials managed?
What unit of work should recover independently?
How much existing infrastructure would migration replace versus duplicate?

The best orchestrator is the one that makes safe behavior obvious and unsafe behavior awkward.

A production checklist for retryable pipelines

Before enabling automatic retries, verify:

Every retryable step has a finite attempt limit.
Retry delays are appropriate for the dependency.
Large worker fleets cannot create synchronized retry storms.
Permanent errors can bypass pointless retries.
Database writes have uniqueness or convergence semantics.
External write APIs use stable idempotency keys when available.
A worker can die immediately after a commit without creating duplicate state on the next attempt.
Repeated execution produces the same final business result.
Retry monitoring distinguishes ordinary transient recovery from a persistent failing dependency.

Then perform the test that exposes most hidden problems:

Run the exact same partition twice.

After the second successful execution, the business state should be equivalent to the state after the first.

A production checklist for backfillable pipelines

Before allowing historical replay, verify:

Every historical run receives an explicit partition or interval.
Pipeline code does not secretly depend on the current wall clock.
Source data for the historical interval still exists.
Historical processing uses the normal transformation code path.
Destination writes are upserts, partition replacement, immutable keyed writes, or another deliberately replay-safe strategy.
Downstream external side effects have been audited separately.
Backfill concurrency has an explicit limit.
Source and destination capacity have been estimated.
Historical compute cost has been estimated.
Live scheduled workloads retain enough capacity.
You know whether partitions should run oldest-first or newest-first.
Every rebuilt partition has a validation signal.
A partially completed backfill can be resumed safely.
You know how to cancel the operation without leaving ambiguous business state.

If any of those questions produce “we think so,” treat that uncertainty as part of the backfill design work.

The three-run idempotency test

Before calling a pipeline production-ready, run this deliberately.

First, materialize one partition.

Second, materialize exactly the same partition again.

Third, materialize it a third time.

Then compare:

Row counts.
Business totals.
Unique-key counts.
Files or object keys.
External API actions.
Notifications.
Downstream event counts.

The final business state should match the state produced by one correct execution.

If the second or third run changes the result merely because another execution occurred, the pipeline is not replay-safe.

That means retries are dangerous.

And if retries are dangerous, backfills are even more dangerous because they apply the same flaw at historical scale.

From scheduled scripts to an operable system

Cron is still the right tool for many simple jobs.

A command that cleans temporary files every night may not need a workflow orchestrator.

The transition happens when you need to reason about execution rather than merely start execution.

Once your questions become:

What failed?
What succeeded before it failed?
Which dependency blocked this task?
Which partition does this run represent?
Should this particular failure retry?
Can this date safely execute again?
How do we regenerate 60 historical partitions?
How many can we run concurrently?
Did the rebuilt data actually become correct?

you have moved beyond scheduling.

You are operating a data system.

The key lesson is not “replace cron with Airflow” or “use Dagster.”

It is to design computation around explicit runs, deterministic partitions, bounded retries, idempotent effects, controlled historical replay, and validation.

The orchestrator then becomes enormously powerful because repeating work is no longer frightening.

Take one cron-driven daily pipeline and refactor it so its business date is explicit, its destination has a stable uniqueness or partition key, its writes are replay-safe, and a worker can crash immediately after committing without corrupting the next attempt. Then backfill a week that already exists and prove that every partition still converges to one correct result.

Try this exercise on one cron-driven pipeline this week, then continue to the next lesson on observability and alerting.

Cron tells you when. An orchestrator tells you what happened.

A DAG is more than a prettier crontab

The most important object is the run

Retries are safe only when the work is replayable

1. Deterministic input boundaries

2. Idempotent side effects

3. Bounded retries

Build a runnable Airflow 3.3.1 retry experiment

Run the Airflow experiment

Retry state is not business state

Backfills solve the problem cron never really modeled

A 365-day backfill is 365 production workloads

Catchup and backfill are related, but different

Logical time beats wall-clock time

Idempotency means convergence, not merely suppressing errors

Side effects deserve stricter treatment than data transforms

Dagster 1.13.18 starts from a different mental model

One partition per run, batches, or one giant run?

One run per partition

Batched partitions

Single-run backfill

Step retries and whole-run retries are different tools

The cherry on the cake: a backfill that doubled revenue

Backfills should be boring

Validation must check data, not just scheduler state

Plan for backfill amplification

Design retries around the failure you actually expect

Transient infrastructure failure

Rate limiting

Permanent configuration failure

Bad business input

Airflow or Dagster?

Airflow is natural when the workflow is the primary object

Dagster is natural when data assets are the primary object

A production checklist for retryable pipelines

A production checklist for backfillable pipelines

The three-run idempotency test

From scheduled scripts to an operable system

Vulnerability Watch No52

Tech Layoffs Watch No13

Orchestration beyond cron: DAGs, retries, and backfills with Airflow or Dagster