Orchestration in 2026: Airflow assets and dynamic task mapping vs AWS Step Functions and Lambda, and what to pick for data vs ML

Modern workflow orchestration is no longer just about scheduling a Python script at midnight. Today’s data platforms must coordinate warehouse transformations, react to incoming files, handle failures across distributed services, launch machine learning training jobs, and process thousands or millions of independent records without losing track of progress.

Two particularly important approaches in 2026 are Apache Airflow 3.x, with Python-defined workflows, dynamic task mapping, and asset-aware scheduling, and AWS Step Functions with Lambda, with fully managed state machines, event-driven execution, and serverless compute.

Both can coordinate dependencies, parallelism, retries, and failure recovery. But their execution models, economics, and operational responsibilities differ significantly.

The right choice depends on what you are orchestrating:

• A daily warehouse transformation is fundamentally different from an event-driven file ingestion pipeline.

• A machine learning training job has different resource and duration requirements from a Lambda function.

• Processing 20 files is different from processing 20 million records.

• A workflow that must survive a week-long approval process needs different state management from a task that finishes in five seconds.

This guide covers the current APIs and capabilities as of October 16, 2026.

By the end, you will understand how to build and operate workflows with Airflow, compare the wider orchestration landscape, test Lambda functions locally, deploy an AWS Step Functions state machine, and choose an appropriate orchestrator for data engineering and ML.

Two practical labs accompany the explanations. The Airflow lab runs on a laptop without a cloud account. The AWS lab separates credential-free Python testing from cloud validation and deployment.

All example data is synthetic retail data.

Part 1: Apache Airflow 3.x — a modern data workflow orchestrator

What version of Airflow should you use in October 2026?

The latest released Apache Airflow version as of October 16, 2026, is 3.3.2, released September 17, 2026.

This guide targets Airflow 3.3.2, Python 3.12, and the public Task SDK.

The differences from Airflow 2.x are substantial.

Airflow 3.2 introduced native asset partitioning, while Airflow 3.3 expanded it and added state stores, retry-policy extensibility, and additional language support.

These are real changes to the platform, not cosmetic renaming.

The most important recommendation for new Python workflows is to use the public Task SDK rather than depending on internal Airflow modules.

References: Airflow 3.3.2 release notes and Airflow 3.3 announcement.

The Airflow mental model: DAGs, tasks, operators, and executors

A DAG, or Directed Acyclic Graph, describes operations and the dependencies between them.

Imagine a retail analytics workflow:

Discover daily order files
        |
        +--> Validate file A --+
        +--> Validate file B --+   (one mapped task per file)
        +--> Validate file C --+
                               |
                               v
Publish clean data and manifest  -> updates an asset
                               |
                               v
Downstream reporting DAG runs (scheduled on that asset)

Conceptual flow: mapped processing tasks publish a data asset consumed by another DAG.

The graph is the workflow definition. Airflow coordinates its execution.

The important components are:

• DAG: A workflow definition containing tasks, dependencies, scheduling rules, and configuration.

• Task: A unit of work. Each execution has a task instance associated with a particular DAG run.

• Operator: A reusable implementation for performing work, such as executing Python, running SQL, or starting a cloud job.

• Scheduler: Determines when DAG runs and task instances are eligible to execute.

• Executor: Determines how task instances run.

• DAG processor: Parses DAG code and prepares serialized definitions for the orchestration system.

• Metadata database: Stores scheduling state, task history, and other metadata.

• API server: Exposes the REST API and serves the user interface.

Executors determine where work happens. LocalExecutor runs tasks on the local machine. CeleryExecutor distributes work to worker processes, while KubernetesExecutor runs tasks in Kubernetes pods.

An essential architectural principle is that Airflow does not have to execute all processing inside its workers.

For example, an Airflow task can launch a Spark job on an external cluster, capture its job identifier, and monitor completion. The orchestrator coordinates the operation; the compute platform performs the expensive calculation.

Operators versus TaskFlow

Airflow supports two complementary programming styles.

With traditional operators, you instantiate task objects, configure them, and explicitly define dependencies.

TaskFlow uses decorated Python functions:

• @dag defines a DAG factory.

• @task turns a Python function into an Airflow task.

• Calling decorated functions while constructing the DAG creates task dependencies.

• Passing a task’s output to another task connects them automatically.

TaskFlow is especially convenient for Python-heavy pipelines.

Operators remain useful for external services. A mature database or AWS operator may already implement authentication, templating, retries, and asynchronous waiting.

There is no reason to replace a well-supported operator with handwritten API calls solely to make everything look like Python functions.

XCom: pass references, not large datasets

XCom, short for cross-communication, is how Airflow tasks exchange small serialized values.

A TaskFlow function can return a list, dictionary, filename, object-storage URI, or job ID. Airflow stores the result so downstream tasks can access it.

Appropriate XCom values include:

• A list of discovered filenames.

• An S3 URI for a generated manifest.

• A remote training job ID.

• A row count or validation result.

• A short collection of partition identifiers.

An inappropriate XCom value is a multi-gigabyte DataFrame.

Large data belongs in object storage, a warehouse, a filesystem accessible to workers, or another data platform. XCom should contain a reference to that data.

Keeping bulk data out of orchestration metadata reduces unnecessary serialization, database pressure, and failure recovery costs.

Retries and idempotency: why successful workflows can still duplicate data

Suppose an Airflow task appends 10,000 retail orders to a warehouse.

The warehouse accepts the write, but the worker loses its connection before Airflow records success.

Airflow retries.

If the code blindly appends again, the warehouse now contains 20,000 rows.

The retry mechanism worked exactly as configured, but the data is wrong.

This is why idempotency matters more than retries alone.

An idempotent operation produces the same intended result when repeated with the same logical input.

Useful techniques include:

• Upserting rows using stable order identifiers.

• Replacing a specific date partition rather than appending indefinitely.

• Publishing immutable, versioned output objects.

• Writing temporary output followed by an atomic rename where supported.

• Recording event IDs to detect duplicates.

• Using idempotency keys for external APIs that support them.

Design every retryable task around the possibility that the previous attempt succeeded remotely even though the orchestrator observed a failure.

Sensors versus deferrable operators

Sensors wait for a condition.

For example, a pipeline might need to wait until an upstream system publishes an object in S3.

A conventional polling sensor in poke mode occupies a worker slot while it waits.

A sensor in reschedule mode releases its worker slot between checks.

A deferrable operator goes further: it suspends its task while Airflow’s triggerer handles asynchronous waiting, then resumes when the condition is satisfied.

Deferrable operators are useful when waiting for long-running external jobs, storage events, or other conditions that do not require active CPU execution.

Deferral requires appropriate operator and trigger support. It does not automatically turn arbitrary blocking Python code into asynchronous work.

Pools, concurrency, catchup, and backfills

Airflow provides several controls to prevent workflows from overwhelming external systems.

Pools limit how many task slots can concurrently use a constrained resource. A pool with five slots can protect a database from 100 independently scheduled workers trying to connect at once.

Other useful controls include task concurrency limits, maximum active DAG runs, and executor capacity.

Catchup determines whether Airflow creates previously unscheduled historical runs between a DAG’s start date and the current scheduling interval.

When catchup is disabled, a newly enabled daily DAG ordinarily begins with its current applicable schedule rather than automatically creating its entire historical backlog.

A backfill is an intentional replay of historical processing intervals.

For example, suppose a transformation bug affected seven days of sales reporting. You could fix the code, backfill those seven logical dates, and safely replace their affected partitions.

Airflow 3 uses scheduler-managed backfills with explicit replay behavior, rather than requiring older Airflow 2.x backfill practices.

A backfill is only safe if the output design makes reruns predictable. Otherwise, replaying data can create duplicates, overwrite newer results, or mix incompatible transformation versions.

Reference: Airflow 3.3 backfill documentation.

Dynamic task mapping: the array of tasks created at runtime

Dynamic task mapping is one of Airflow’s most practical capabilities for partitioned data processing.

Suppose an upstream step discovers 500 new files.

You do not know their names until the workflow runs, so defining 500 fixed tasks in advance is inappropriate.

Dynamic mapping lets an upstream task return a list, then creates a separate downstream task instance for each element.

Two methods define the behavior:

• .expand() supplies the values that vary between task instances.

• .partial() supplies arguments shared by all mapped instances.

For instance, one task might return three filenames. The next task expands into three mapped instances, with independent states, logs, and retries.

Dynamic mapping does not provide unlimited parallelism. Pools, scheduler settings, available workers, and downstream service capacity still apply.

There is also an important granularity decision.

Creating 500 mapped tasks for 500 files is often reasonable. Creating five million mapped tasks to process five million tiny records may impose excessive scheduling overhead.

For extremely large datasets, process chunks or partitions instead.

Assets: data-aware scheduling instead of only clock-based scheduling

Airflow 2.4 introduced data-aware scheduling through Datasets. Airflow 3 renamed the concept to Assets.

An asset represents a logical data resource, such as a published table, object-storage object, or processed dataset.

A producer task declares that it updates an asset. A consumer DAG declares that the asset drives its schedule.

The lifecycle works as follows:

1. An upstream task performs its transformation.

2. The task finishes successfully and emits an asset update.

3. Airflow records the asset event.

4. The scheduler determines which downstream DAGs are eligible.

5. The consumer DAG runs.

An asset is not automatically a filesystem watcher. Airflow records an asset update because an appropriately configured producing task successfully finished, not because it independently inspected every physical object.

Multiple assets can be combined into scheduling conditions. For example, a reporting DAG may require updates to both orders and product catalog data.

Newer Airflow releases also support richer asset partitioning, making data-aware scheduling more useful for independently updated daily or hourly partitions.

Reference: Airflow asset-aware scheduling.

Hands-on lab 1: two Airflow DAGs running on your laptop

Let’s implement a complete workflow without an AWS account.

The producer DAG will generate three small retail CSV files, discover their filenames, dynamically map file processing across them, publish a JSON manifest, and emit an asset event.

The consumer DAG will be scheduled by that asset and read the manifest.

Step 1: Install Airflow 3.3.2

Use Python 3.12 on Linux, macOS, or Windows Subsystem for Linux 2. Native Windows installation is not the supported path for Apache Airflow.

Create the project:

mkdir -p orchestration-lab
cd orchestration-lab

python3.12 -m venv .venv
source .venv/bin/activate

export AIRFLOW_HOME="$(pwd)/.airflow"
mkdir -p "$AIRFLOW_HOME/dags"

Install the exact Airflow release with its corresponding constraints file:

python -m pip install \
  "apache-airflow==3.3.2" \
  --constraint \
  "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.12.txt"

airflow db migrate

The constraints file resolves a compatible dependency set for the selected Airflow and Python versions.

Avoid installing Airflow with an arbitrary combination of latest transitive dependencies. Airflow’s own constraints are the recommended basis for reproducible installations.

This local lab uses SQLite and requires no cloud credentials.

Step 2: Create the producer and consumer DAGs

Save the following file as .airflow/dags/retail_pipeline.py.

import csv
import json
import os

from datetime import datetime, timedelta, timezone
from pathlib import Path

from airflow.sdk import Asset, dag, task


HOME = Path(os.environ["AIRFLOW_HOME"])
DATA_DIR = HOME / "retail_demo"

ORDERS_READY = Asset("retail://demo/orders_ready")


@dag(
    dag_id="retail_file_ingest",
    start_date=datetime(2026, 1, 1, tzinfo=timezone.utc),
    schedule=None,
    catchup=False,
    max_active_runs=1,
    tags=["retail", "dynamic-mapping"],
)
def retail_file_ingest():

    @task
    def discover_files() -> list[str]:
        incoming = DATA_DIR / "incoming"
        incoming.mkdir(parents=True, exist_ok=True)

        samples = {
            "orders_a.csv": "id,amount\n1,15\n2,25\n",
            "orders_b.csv": "id,amount\n3,40\n4,60\n",
            "orders_c.csv": "id,amount\n5,10\n",
        }

        for name, contents in samples.items():
            (incoming / name).write_text(
                contents,
                encoding="utf-8",
            )

        return sorted(samples)

    @task(
        retries=2,
        retry_delay=timedelta(seconds=5),
    )
    def process_file(
        filename: str,
        prefix: str,
    ) -> dict:
        source = DATA_DIR / "incoming" / filename

        target_dir = DATA_DIR / "processed"
        target_dir.mkdir(parents=True, exist_ok=True)

        target = target_dir / f"{prefix}{filename}"
        temporary = target.with_suffix(".tmp")

        with source.open(
            newline="",
            encoding="utf-8",
        ) as inp:
            rows = list(csv.DictReader(inp))

        with temporary.open(
            "w",
            newline="",
            encoding="utf-8",
        ) as out:
            writer = csv.DictWriter(
                out,
                fieldnames=["id", "amount"],
            )
            writer.writeheader()

            for row in rows:
                writer.writerow({
                    "id": int(row["id"]),
                    "amount": int(row["amount"]),
                })

        temporary.replace(target)

        return {
            "file": filename,
            "rows": len(rows),
            "output": str(target),
        }

    @task(outlets=[ORDERS_READY])
    def publish_manifest(
        results: list[dict],
    ) -> dict:
        sorted_results = sorted(
            results,
            key=lambda item: item["file"],
        )

        manifest = {
            "files": sorted_results,
            "total_rows": sum(
                item["rows"]
                for item in sorted_results
            ),
        }

        target = (
            DATA_DIR
            / "processed"
            / "manifest.json"
        )

        temporary = target.with_suffix(".tmp")

        temporary.write_text(
            json.dumps(manifest, indent=2),
            encoding="utf-8",
        )
        temporary.replace(target)

        return {
            "manifest": str(target),
            "total_rows": manifest["total_rows"],
        }

    files = discover_files()

    processed = (
        process_file
        .partial(prefix="clean_")
        .expand(filename=files)
    )

    publish_manifest(processed)


@dag(
    dag_id="retail_asset_consumer",
    start_date=datetime(2026, 1, 1, tzinfo=timezone.utc),
    schedule=[ORDERS_READY],
    catchup=False,
    tags=["retail", "assets"],
)
def retail_asset_consumer():

    @task
    def report() -> None:
        path = (
            DATA_DIR
            / "processed"
            / "manifest.json"
        )

        manifest = json.loads(
            path.read_text(encoding="utf-8")
        )

        print(
            f"Consumed {manifest['total_rows']} "
            f"orders from {len(manifest['files'])} files"
        )

    report()


producer = retail_file_ingest()
consumer = retail_asset_consumer()


if __name__ == "__main__":
    producer.test()

Step 3: Understand what the code is doing

The discover_files() function generates the synthetic source data and returns three filenames.

That list becomes the mapping input.

The following two calls are the important part of the example:

• .partial(prefix="clean_") binds a common output filename prefix.

• .expand(filename=files) creates a mapped task instance for every discovered filename.

Each mapped task reads a CSV, converts the identifier and amount values to integers, and writes a cleaned CSV.

The output is written to a temporary file and then replaced. That is more retry-friendly than continually appending to the destination.

After all mapped tasks succeed, publish_manifest() collects the outputs, sorts them, calculates the total row count, and writes a JSON manifest.

Because this task declares outlets=[ORDERS_READY], successful completion also publishes an asset update.

The second DAG subscribes through schedule=[ORDERS_READY].

There is no scheduled midnight timestamp in the consumer. The publication event is the scheduling dependency.

Step 4: Test the producer with dag.test()

Airflow 3.3 requires an initialized metadata database and a DAG file available through a configured DAG bundle for dag.test() to execute its real DAG run.

The installation and file placement above meet those requirements.

Run:

python "$AIRFLOW_HOME/dags/retail_pipeline.py"

The script’s main guard invokes producer.test().

The expected producer execution is:

• One discovery task.

• Three mapped process_file task instances.

• One manifest publication task.

Inspect the generated manifest:

cat "$AIRFLOW_HOME/retail_demo/processed/manifest.json"

The manifest will contain three file records and a total row count of five. Output paths are absolute paths under your own Airflow home directory.

You can verify the result automatically:

python - <<'PY'
import json
import os
from pathlib import Path

path = (
    Path(os.environ["AIRFLOW_HOME"])
    / "retail_demo"
    / "processed"
    / "manifest.json"
)

manifest = json.loads(
    path.read_text(encoding="utf-8")
)

assert len(manifest["files"]) == 3
assert manifest["total_rows"] == 5
assert sorted(
    item["rows"] for item in manifest["files"]
) == [1, 2, 2]

for item in manifest["files"]:
    assert Path(item["output"]).is_file()

print("Producer output verified: 3 files, 5 rows")
PY

This checks the actual output artifact, rather than merely assuming task completion means the data is correct.

Step 5: Run Airflow standalone and verify the asset consumer

Start Airflow’s local standalone environment:

airflow standalone

Leave that terminal running.

In a second terminal, return to the project and activate the same environment:

cd orchestration-lab
source .venv/bin/activate

export AIRFLOW_HOME="$(pwd)/.airflow"

airflow dags list
airflow dags list-import-errors

Ensure that both DAGs appear and that there are no import errors.

Unpause both DAGs:

airflow dags unpause retail_file_ingest
airflow dags unpause retail_asset_consumer

Trigger the producer:

airflow dags trigger retail_file_ingest

Open http://localhost:8080 and inspect the two DAGs.

The standalone installation generates local authentication credentials. The startup output and local password file provide the relevant login details.

The verification sequence is:

1. Open retail_file_ingest and inspect the mapped task instances.

2. Confirm the producer’s manifest task succeeds.

3. Inspect the asset event for retail://demo/orders_ready.

4. Open retail_asset_consumer.

5. Confirm the consumer completes and logs that it consumed five orders from three files.

The earlier dag.test() execution may already have recorded an asset event. Depending on when the consumer becomes active, you may therefore observe an additional consumer run. Compare the latest producer event with its downstream run rather than assuming exactly one consumer run exists.

This is an end-to-end verification procedure for the scheduler-driven dependency, not simply a standalone Python function test.

Verification scope: The Python syntax of the example was checked, but a live Airflow 3.3.2 standalone run could not be performed in the validation environment used to prepare this guide. The expected task and event behavior follows the documented Airflow 3.3.2 APIs; the commands above are the reproducible integration check to run on the target installation.

Reference: Airflow 3.3 debugging and dag.test()documentation.

Lessons from the Airflow lab

The example contains several production patterns worth retaining.

Store data outside XCom. The mapped tasks return small dictionaries rather than the full content of their CSV files.

Make output publication retry-safe. The example replaces output files instead of continually appending to them.

Represent dependencies through assets. The reporting DAG depends on successful data publication, not a guessed delay after an upstream schedule.

Control overlapping runs. The producer limits active runs to one because its output paths are shared. A production deployment should also use partition-specific or run-specific paths when concurrency is required.

Use shared durable storage in production. A local filesystem works because the scheduler and tasks share one laptop. Distributed workers may not have access to the same files. Replace local paths with object storage, a warehouse, or a properly managed shared filesystem.

Finally, distinguish the orchestration event from data correctness. A task may finish successfully while producing unexpected output. Validate schemas, row counts, freshness, and other quality requirements before publishing the asset.

Part 2: The wider orchestration landscape

Airflow is only one member of a much larger workflow orchestration ecosystem.

Different orchestrators make different assumptions about what a workflow represents.

Some treat a workflow as a graph of scheduled tasks. Others treat data assets as the primary abstraction. Some are designed for durable business processes, while others focus on reproducible ML computation.

Eight orchestration tools worth understanding

These products are not strictly interchangeable.

For example, Temporal is especially strong for a business process that waits three days for customer approval and must survive service restarts.

Dagster is attractive when teams organize their platform around published data products and their lineage.

Kubeflow Pipelines and Flyte can be preferable when the dominant requirements are containerized compute, ML dependencies, GPUs, and reproducible execution environments.

Metaflow provides a relatively approachable transition from local Python experimentation into infrastructure-backed workflows.

The best system is the one whose abstraction matches the workload, rather than the one with the largest feature list.

Managed Airflow: Amazon MWAA and Cloud Composer

Operating Airflow requires a scheduler, API server, DAG processor, metadata database, executors, logging, and deployment processes.

Managed services reduce this operational burden.

Two important choices are:

• Amazon Managed Workflows for Apache Airflow (MWAA): Managed Airflow infrastructure integrated with AWS.

• Google Cloud Composer: Managed Airflow integrated with Google Cloud.

These services are particularly relevant when an organization already uses Airflow but does not want to operate the control plane itself.

Managed Airflow is not operations-free. Teams still own DAG design, dependencies, data correctness, resource tuning, task retries, and observability.

Managed service releases can also lag behind upstream Apache Airflow. Before adopting an Airflow 3.3 feature, verify that the specific managed offering supports the required Airflow release.

For a large, cross-system data platform, managed Airflow may be well worth its persistent infrastructure cost.

For a new AWS-only application that handles a handful of daily events, Step Functions may be simpler.

Part 3: AWS serverless orchestration with Lambda and Step Functions

Airflow generally runs as a persistent orchestration platform.

AWS Step Functions provides a different model: AWS operates the state-machine engine, and you define the workflow states, transitions, integrations, and error handling.

Lambda supplies event-driven computation. Step Functions coordinates the larger process.

The two services complement one another, but AWS’s newer Lambda execution options create additional architectural choices in 2026.

AWS Lambda fundamentals and execution limits

AWS Lambda executes code in response to events, messages, schedules, or direct API calls.

A typical Python handler accepts an event and context object, performs a unit of work, and returns a JSON-serializable result.

Useful data engineering applications include:

• Validating an uploaded CSV file.

• Reading S3 object metadata.

• Normalizing a small batch of retail events.

• Submitting an external processing job.

• Updating a DynamoDB metadata record.

• Enriching a queue message.

• Invoking an existing inference endpoint.

Lambda is most attractive for short, independent workloads with variable arrival rates.

For standard on-demand Lambda functions, the important limits are:

These are the default Lambda compute limits. Lambda Managed Instances have additional capabilities discussed shortly.

The /tmp directory can hold intermediate files, downloaded artifacts, or decompressed archives. However, it is not durable storage. Never depend on a later invocation using the same execution environment.

Use S3, a database, or another durable system for important output.

References: Lambda quotas, memory configuration, and timeout configuration.

Container images, response streaming, and SnapStart

Lambda supports both ZIP and container-image packaging.

Container images are valuable when functions require native dependencies, larger library collections, or complex runtime packaging.

A container image does not automatically extend Lambda’s invocation timeout. It changes packaging, not the basic invocation contract.

Lambda response streaming allows eligible integrations to receive response data incrementally, rather than waiting for the complete payload.

AWS supports streamed responses up to 200 MB, compared with 6 MB for standard buffered synchronous responses.

There is an important Python-specific distinction: Python managed runtimes do not expose the same native response-streaming handler interface as Node.js. Python applications can use a custom runtime or Lambda Web Adapter for streaming.

Streaming is useful for interactive responses. It is not a substitute for sending large analytical datasets to S3.

SnapStart addresses a different concern: initialization latency.

SnapStart captures an initialized execution environment and restores it for eligible function versions, reducing repeated initialization work.

It supports eligible Python managed runtimes, including Python 3.12 and newer supported versions.

A notable 2026 update is that SnapStart also supports Lambda container-image functions, announced September 2, 2026. This can be useful for latency-sensitive Python inference functions with expensive startup requirements.

Snapshot-aware initialization must handle credentials, unique identifiers, and other time-sensitive state carefully.

References: Lambda response streaming and SnapStart for container images.

Newer Lambda option 1: durable functions

AWS announced Lambda durable functions on December 2, 2025.

Durable functions allow a logical application workflow to survive interruptions and span multiple invocations using checkpoints, waits, and replay.

An asynchronous durable execution can run for up to one year.

Consider a retail order fulfillment process:

1. Validate the order.

2. Reserve inventory.

3. Wait for supplier confirmation.

4. Arrange fulfillment.

5. Record the final outcome.

A traditional Lambda invocation cannot simply remain active for several days. A developer would have to persist progress and arrange later invocations, or use an external workflow engine.

Durable functions provide this behavior within the Lambda programming model.

The Python Durable Execution SDK includes:

• @durable_execution to mark a durable handler.

• @durable_step to declare checkpointed steps.

• DurableContext to manage durable operations, waits, and workflow progress.

When a function resumes, the runtime replays recorded execution history and substitutes checkpointed results for completed steps.

That introduces an important programming constraint: replayed code must be deterministic, and external side effects should be performed inside appropriately managed durable steps.

Random values, current timestamps, mutable external state, and untracked network calls can cause problems when code is replayed.

The one-year duration limit needs a qualification

A durable execution lasting one year does not mean one Lambda invocation can execute CPU-bound Python for a year.

Individual invocations still have their timeout limits.

The invocation mode also matters:

For a very long workflow initiated by an SQS event-source mapping, AWS documents an intermediary-function pattern: the mapping invokes a short-lived function, which starts the durable function asynchronously.

This avoids accidentally holding the event-source mapping open throughout a multi-day process.

References: Durable function invocation behavior and event-source mapping limits.

When to choose durable Lambda

Durable functions are attractive when the workflow primarily consists of application-specific logic and sequential recoverable steps.

They can beat a plain Lambda handler when steps involve long waits, external approvals, checkpointing, or recovery from interruption.

They can beat Step Functions when the development team prefers a code-centric workflow and does not need extensive state-machine-level coordination across AWS services.

Step Functions remains attractive when the workflow’s primary responsibility is starting and monitoring Glue, SageMaker, ECS, Batch, or other managed jobs.

Durable execution also has its own charging dimensions, including operations and retained execution state. Check AWS Lambda pricing rather than assuming suspended compute means the complete workflow is free.

Newer Lambda option 2: Lambda Managed Instances

Lambda Managed Instances changes the infrastructure and economics of Lambda execution.

Instead of relying exclusively on Lambda’s default on-demand execution environments, functions run on Lambda-managed EC2 capacity in your AWS account.

AWS manages supported infrastructure lifecycle tasks while exposing an instance-based capacity model.

The important capabilities include:

• EC2 instance-type selection.

• Multiple concurrent invocations within supported execution environments.

• Configurable memory-to-vCPU ratios.

• Up to 32 GB of memory and 16 vCPUs, introduced March 27, 2026.

• Up to 90 minutes per supported asynchronous or event-source-mapping invocation.

Synchronous invocations remain limited to 15 minutes. The 90-minute timeout was announced on September 9, 2026. Amazon MQ and Amazon DocumentDB event-source mappings are exceptions to the extended 90-minute limit.

The pricing model also differs from standard Lambda.

AWS documents EC2-based instance charges, Lambda request charges, and a management fee equal to 15% of the underlying EC2 on-demand instance price. EC2 discounts can reduce the instance portion, but not the management fee.

Capacity can remain billable while no application requests are executing.

For sustained high-throughput workloads, Managed Instances may provide better economics than the default model.

For infrequent bursts, standard Lambda often remains more attractive.

References: Managed Instances documentation, 32 GB and 16 vCPU announcement, and 90-minute invocation guidance.

Choosing the Lambda execution model

Durable functions and Managed Instances solve different problems and are not inherently competing abstractions.

A durable function improves workflow-state management. Managed Instances change the compute capacity model.

Neither option automatically makes Lambda the best environment for GPU training, large Spark transformations, or distributed model tuning.

Those workloads generally belong on dedicated compute platforms.

Step Functions: orchestrating work with state machines

Step Functions uses Amazon States Language (ASL), a JSON-based state-machine definition.

A state machine contains states, transitions, input/output transformations, and error-handling rules.

The core state types

Step Functions makes workflow transitions and failure behavior explicit.

Consider a file ingestion process:

• An object arrival starts the workflow.

• One state validates the object metadata.

• A Choice state rejects unsupported formats.

• A Task state submits the transformation.

• A later state publishes results.

• Failure handlers record relevant errors.

The state machine coordinates the work without requiring a persistent scheduler cluster.

Standard versus Express workflows

AWS provides two major Step Functions workflow types.

The exactly-once terminology for Standard workflows describes Step Functions’ execution semantics.

It does not eliminate the possibility of duplicate effects in external systems.

Explicit retries, lost acknowledgements, and downstream service behavior still require idempotent task design.

Standard workflows are usually the better default for long-running data pipelines, ML training coordination, human approvals, and other operations where durable execution history matters.

Express workflows are attractive for large volumes of short, repeatable events.

Reference: AWS Step Functions workflow types.

Inline Map versus Distributed Map

A Map state is the Step Functions equivalent of runtime fan-out.

Like Airflow dynamic task mapping, it accepts a collection and executes work for each element.

But its execution model differs.

Inline Map processes a JSON array within the parent workflow execution. It supports up to 40 concurrent iterations.

This is a good fit for processing tens of files or making a modest number of independent service calls.

Distributed Map launches child workflow executions and supports up to 10,000 concurrent child executions.

It can read supported data sources directly from S3, including JSON arrays, CSV files, object lists, and S3 inventory data.

Use Distributed Map when the parent execution’s 25,000-event history limit, payload size limit, or Inline Map concurrency limit would otherwise become a problem.

For millions of tiny records, batching is still important. A million individual child executions can generate considerable orchestration overhead and cost.

Reference: AWS Step Functions Map state documentation.

Retry and Catch: separate transient failures from permanent failures

Step Functions provides retry and error handling directly in ASL.

A Retry policy specifies eligible errors, retry count, interval, backoff, and optional jitter.

A Catch rule routes an error to another state after the eligible retries are exhausted.

For example, a Lambda throttling error may justify retrying with exponential backoff.

An invalid CSV filename does not improve after waiting five seconds. It should normally fail validation and be handled as a permanent input error.

Retries should target specific transient failures rather than indiscriminately retrying everything.

There are also terminal runtime errors that a general States.ALL catcher does not intercept, so validate the workflow input structure and payload sizes separately.

Direct integrations: fewer Lambda wrappers

A common serverless design mistake is writing a Lambda function whose entire purpose is to call one AWS API.

Step Functions often supports that API directly.

Useful integrations include:

• AWS Glue: Start and monitor ETL jobs.

• Amazon EMR Serverless: Coordinate Spark processing.

• AWS Batch: Launch batch compute jobs.

• Amazon ECS and Fargate: Run container workloads.

• Amazon SageMaker AI: Coordinate training, transformation, and other ML jobs.

• Amazon Bedrock: Invoke supported model and AI-related operations.

Two integration patterns are especially important.

Run a Job (.sync) starts a supported external operation and waits for it to finish.

Wait for Callback (.waitForTaskToken) suspends progression until an external actor completes the task through the callback API.

A callback token is a sensitive completion capability. Protect it with appropriate IAM permissions and timeout controls.

Not every service action supports every integration pattern. Check the individual integration documentation before designing a workflow around synchronous completion or callbacks.

Reference: Step Functions integration patterns.

Event plumbing: EventBridge, S3, SQS, and Pipes

An orchestrator also needs a way to know when work should start.

A common AWS-native ingestion architecture is:

S3 upload → EventBridge → Step Functions → file validation → Glue ETL → published output.

For higher arrival rates, SQS can provide buffering and backpressure.

A CDC pipeline might use a database change stream, Kinesis or SQS, and an appropriate consumer that updates a downstream system.

Avoid triggering a long-running Standard state machine separately for every database row when records should instead be batched or processed continuously.

Event-driven does not automatically mean exactly-once delivery. S3 event notifications, standard SQS queues, and other distributed event services may produce duplicate deliveries.

Use stable object identifiers, source offsets, event IDs, or application-defined idempotency keys to protect downstream effects.

Hands-on lab 2: Python Lambda and a Step Functions state machine

Now let’s implement a deployable serverless example.

The workflow will accept a JSON array of retail CSV filenames, process each element through Inline Map, invoke a Python Lambda handler, retry transient Lambda failures, and return the accepted filenames.

The project separates local validation from cloud deployment.

Local Python tests, JSON checks, and SAM template building do not require AWS credentials. AWS state-machine validation and deployment do require an AWS account and appropriate permissions.

Step 1: Create the project

Use Python 3.12 and pin the relevant Python tools:

• pytest 9.1.1, released June 19, 2026.

• AWS SAM CLI 1.169.0, released October 13, 2026.

These release versions were checked against PyPI.

References: pytest 9.1.1 and AWS SAM CLI 1.169.0.

Create the project:

mkdir -p serverless-lab/src serverless-lab/tests
cd serverless-lab

python3.12 -m venv .venv
source .venv/bin/activate

python -m pip install \
  "pytest==9.1.1" \
  "aws-sam-cli==1.169.0"

touch src/__init__.py

The project structure will be:

serverless-lab/
├── .venv/
├── src/
│   ├── __init__.py
│   └── app.py
├── tests/
│   └── test_app.py
├── workflow.asl.json
└── template.yaml

The Lambda runtime code uses only Python’s standard library.

Step 2: Write the Lambda handler

Save the following as src/app.py.

import re


CSV_NAME = re.compile(
    r"[A-Za-z0-9][A-Za-z0-9._-]*\.csv\Z"
)


def lambda_handler(
    event: dict,
    context: object,
) -> dict:
    filename = event.get("filename")

    if (
        not isinstance(filename, str)
        or not CSV_NAME.fullmatch(filename)
    ):
        raise ValueError(
            "filename must be a simple CSV basename"
        )

    return {
        "filename": filename,
        "status": "accepted",
        "kind": "retail-orders",
    }

This handler validates a simple CSV basename.

It rejects directory separators, hidden-file names beginning with a dot, non-string inputs, and filenames without the required CSV extension.

The regular expression deliberately permits a restricted set of ASCII characters. Real applications may have different filename requirements.

Also, accepting a filename does not mean that a file has been fetched from S3 or that its contents have been validated.

The handler is intentionally limited to one small, testable responsibility.

Step 3: Write the pytest tests

Save the following as tests/test_app.py.

import pytest

from src.app import lambda_handler


def test_accepts_csv():
    result = lambda_handler(
        {"filename": "orders_a.csv"},
        None,
    )

    assert result == {
        "filename": "orders_a.csv",
        "status": "accepted",
        "kind": "retail-orders",
    }


@pytest.mark.parametrize(
    "filename",
    [
        "../orders.csv",
        "notes.txt",
        42,
        ".hidden.csv",
        r"folder\orders.csv",
    ],
)
def test_rejects_invalid_name(filename):
    with pytest.raises(
        ValueError,
        match="simple CSV basename",
    ):
        lambda_handler(
            {"filename": filename},
            None,
        )

Run the tests:

python -m pytest -q

Expected output:

......                                                   [100%]
6 passed

These tests exercise one successful case and five invalid-input cases.

The test suite was executed successfully with six passing tests using the available pytest 9.0.2 environment. The source files also passed Python compilation checks. The requested pytest 9.1.1 pin was verified against PyPI, but an isolated installation of that exact version could not be completed in the test environment because package-network access was unavailable.

The tests use stable pytest APIs and contain no AWS calls, so readers can run the exact pinned configuration on their own computers without an AWS account.

Step 4: Define the Step Functions workflow

Save the following as workflow.asl.json.

This is a complete ASL state-machine definition using the JSONPath query language.

{
  "Comment": "Validate retail file basenames with an Inline Map",
  "StartAt": "ProcessFiles",
  "States": {
    "ProcessFiles": {
      "Type": "Map",
      "ItemsPath": "$.files",
      "MaxConcurrency": 3,
      "ItemSelector": {
        "filename.$": "$$.Map.Item.Value"
      },
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "INLINE"
        },
        "StartAt": "ValidateFile",
        "States": {
          "ValidateFile": {
            "Type": "Task",
            "Resource": "arn:aws:states:::lambda:invoke",
            "Parameters": {
              "FunctionName": "${WorkerArn}",
              "Payload.$": "$"
            },
            "OutputPath": "$.Payload",
            "Retry": [
              {
                "ErrorEquals": [
                  "Lambda.ServiceException",
                  "Lambda.AWSLambdaException",
                  "Lambda.SdkClientException",
                  "Lambda.TooManyRequestsException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 2,
                "BackoffRate": 2.0,
                "JitterStrategy": "FULL"
              }
            ],
            "End": true
          }
        }
      },
      "ResultPath": "$.processed",
      "Catch": [
        {
          "ErrorEquals": [
            "States.ALL"
          ],
          "ResultPath": "$.error",
          "Next": "Failed"
        }
      ],
      "Next": "Completed"
    },
    "Completed": {
      "Type": "Succeed"
    },
    "Failed": {
      "Type": "Fail",
      "Error": "RetailBatchFailed",
      "Cause": "At least one file could not be processed"
    }
  }
}

Step 5: Understand the ASL configuration

The top-level StartAt property makes ProcessFiles the entry state.

ItemsPath tells Step Functions to read the array stored under files.

ItemSelector converts each array element into the event object expected by Lambda.

For example, one string filename becomes an object with a filename field.

The ItemProcessor defines the work performed during every iteration.

Its Lambda Task uses the optimized Lambda service integration.

FunctionName contains a placeholder that AWS SAM will replace with the deployed function ARN.

Payload.$ forwards the iteration input to Lambda.

OutputPath selects the response payload instead of returning the entire Lambda integration result.

The Retry policy handles transient AWS Lambda service failures, including throttling.

It intentionally does not retry the validation function’s ValueError indiscriminately.

After applicable retries, a failure in the Map state follows the Catch transition into the explicit Fail state.

On success, the Map’s output is written under the processed field, preserving the original workflow input.

Using ItemProcessor rather than the older Iterator field follows the current Map definition style.

Step 6: Validate the definition locally

First check that the file contains valid JSON:

python -m json.tool workflow.asl.json > /dev/null

Then perform a structural check with standard Python:

python - <<'PY'
import json
from pathlib import Path

path = Path("workflow.asl.json")

definition = json.loads(
    path.read_text(encoding="utf-8")
)

assert definition["StartAt"] == "ProcessFiles"

mapping = definition["States"]["ProcessFiles"]

assert mapping["Type"] == "Map"
assert mapping["MaxConcurrency"] == 3
assert (
    mapping["ItemProcessor"]["ProcessorConfig"]["Mode"]
    == "INLINE"
)

assert "Retry" in (
    mapping["ItemProcessor"]["States"]["ValidateFile"]
)

assert mapping["Next"] == "Completed"

print("Local JSON and structural checks passed")
PY

These checks are credential-free and verify some useful invariants.

However, they do not provide complete ASL validation. A file can be valid JSON while still containing invalid state-machine properties or transitions.

We will address that with the AWS validation API after defining the deployment template.

Step 7: Write the AWS SAM template

Save this as template.yaml.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Retail filename fan-out using Step Functions and Lambda

Resources:
  ProcessFileFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: app.lambda_handler
      Runtime: python3.12
      Architectures:
        - arm64
      Timeout: 10
      MemorySize: 128

  RetailWorkflow:
    Type: AWS::Serverless::StateMachine
    Properties:
      Type: STANDARD
      DefinitionUri: workflow.asl.json
      DefinitionSubstitutions:
        WorkerArn: !GetAtt ProcessFileFunction.Arn
      Policies:
        - LambdaInvokePolicy:
            FunctionName: !Ref ProcessFileFunction

Outputs:
  WorkflowArn:
    Description: ARN of the state machine
    Value: !Ref RetailWorkflow

The template defines two main resources:

• A Python 3.12 Lambda function.

• A Standard Step Functions state machine.

SAM creates the required execution-role resources and applies the Lambda invocation policy for the state machine.

The DefinitionUri property references the ASL file.

The DefinitionSubstitutions mapping replaces the Lambda ARN placeholder during deployment.

Notice that the project does not need to implement its own Lambda-invocation client. The state machine invokes Lambda directly through the supported integration.

Reference: AWS SAM state-machine resources.

Step 8: Validate and build the SAM application locally

Run:

sam validate --lint
sam build

These steps check the SAM template and prepare the application artifacts.

They do not deploy AWS resources, and AWS credentials are not required for this local build. Use sam validate --lint here: plain sam validate looks up AWS credentials and fails if none are configured yet (you set them up in the next step).

The distinction matters because local Python tests, local template checks, and live AWS integration tests are separate verification layers.

Also, SAM’s local emulation does not provide a complete local Step Functions execution engine. Do not interpret a successful SAM build as proof that an AWS state machine has successfully run.

Step 9: Install and configure AWS CLI v2 for cloud operations

The AWS deployment steps use the aws executable.

AWS SAM CLI and AWS CLI are separate tools. Installing AWS SAM CLI through pip does not guarantee that the AWS CLI v2 executable is installed.

Before continuing, install AWS CLI v2 using the official instructions for your operating system:

AWS CLI v2 installation guide.

Verify the installation:

aws --version

The output should identify AWS CLI version 2.

Use an AWS account and an identity with appropriate permissions. AWS IAM Identity Center with short-lived credentials is preferable where available.

For an IAM Identity Center environment, use the interactive setup:

aws configure sso

For an environment using a different supported credential method, configure the corresponding AWS CLI profile instead.

After authenticating, verify the identity:

aws sts get-caller-identity

This command contacts AWS and therefore requires valid credentials.

Deployment permissions must cover the necessary CloudFormation, IAM, Lambda, Step Functions, and deployment artifact operations.

Do not use the root account or unrestricted long-lived access keys for normal development.

Step 10: Validate ASL using the AWS ValidateStateMachineDefinition API

AWS provides a dedicated API to validate ASL definitions without creating a state machine.

Because our original file contains a SAM substitution placeholder, create a validation copy using a syntactically valid example Lambda ARN.

The example ARN is only a placeholder for structural validation; it does not need to identify an existing function.

python - <<'PY'
from pathlib import Path

source = Path("workflow.asl.json")
target = Path("workflow.validate.asl.json")

definition = source.read_text(
    encoding="utf-8"
)

definition = definition.replace(
    "${WorkerArn}",
    "arn:aws:lambda:us-east-1:"
    "123456789012:function:example",
)

target.write_text(
    definition,
    encoding="utf-8",
)

print("Created validation definition")
PY

Now call the AWS validation API:

aws stepfunctions validate-state-machine-definition \
  --definition file://workflow.validate.asl.json \
  --type STANDARD \
  --severity WARNING \
  --region us-east-1 \
  --query '{result:result,diagnostics:diagnostics}'

A successful validation returns a result of OK. Diagnostics may include warnings even when the definition is valid.

For a CI pipeline, it is useful to make validation failures produce a nonzero command result:

set -o pipefail

aws stepfunctions validate-state-machine-definition \
  --definition file://workflow.validate.asl.json \
  --type STANDARD \
  --region us-east-1 \
  --query result \
  --output text | grep -qx OK

This fails if the API rejects the definition or does not return OK.

The API validates the ASL definition, not the full deployed application. It does not prove that the Lambda exists, that permissions are correct, or that every workflow input succeeds.

It also contacts AWS, so this part cannot be represented as a local-only test.

Verification scope: The ASL file passed local JSON parsing and structural checks. Live AWS ValidateStateMachineDefinition results cannot be claimed without making the authenticated AWS request. The commands above provide that verification step with no state-machine resource creation.

Reference: ValidateStateMachineDefinition API.

Step 11: Deploy with AWS SAM

With AWS CLI v2 configured and an authorized AWS account, run:

sam deploy \
  --stack-name retail-map-demo \
  --resolve-s3 \
  --capabilities CAPABILITY_IAM \
  --region us-east-1 \
  --no-confirm-changeset

This command deploys real AWS resources and can incur charges.

SAM packages the application, uploads deployment artifacts as needed, and creates or updates a CloudFormation stack.

The command uses us-east-1 for consistency with the validation example. Choose an appropriate supported region for your real application.

Once CloudFormation completes successfully, the state machine and Lambda function are available in the selected account and region.

Step 12: Start the deployed workflow

Retrieve the state-machine ARN from the CloudFormation stack outputs:

WORKFLOW_ARN="$(aws cloudformation describe-stacks \
  --stack-name retail-map-demo \
  --region us-east-1 \
  --query "Stacks[0].Outputs[?OutputKey=='WorkflowArn'].OutputValue | [0]" \
  --output text)"

Start an execution with three filenames:

EXECUTION_ARN="$(aws stepfunctions start-execution \
  --state-machine-arn "$WORKFLOW_ARN" \
  --region us-east-1 \
  --input '{"files":["orders_a.csv","orders_b.csv","orders_c.csv"]}' \
  --query executionArn \
  --output text)"

Inspect execution status:

aws stepfunctions describe-execution \
  --execution-arn "$EXECUTION_ARN" \
  --region us-east-1 \
  --query '{status:status,output:output}'

Because Standard workflows start asynchronously, the first response may report RUNNING.

Repeat the inspection after the execution completes.

A successful workflow has a SUCCEEDED status and output containing the original filenames and three processed results.

The successful result is logically equivalent to:

{
  "files": [
    "orders_a.csv",
    "orders_b.csv",
    "orders_c.csv"
  ],
  "processed": [
    {
      "filename": "orders_a.csv",
      "status": "accepted",
      "kind": "retail-orders"
    },
    {
      "filename": "orders_b.csv",
      "status": "accepted",
      "kind": "retail-orders"
    },
    {
      "filename": "orders_c.csv",
      "status": "accepted",
      "kind": "retail-orders"
    }
  ]
}

The example shows the decoded logical output. The CLI’s describe-execution response represents its output field as a serialized JSON string.

You can also inspect the workflow visually in the Step Functions console.

Step 13: Test an error path

Start another execution containing an invalid filename:

aws stepfunctions start-execution \
  --state-machine-arn "$WORKFLOW_ARN" \
  --region us-east-1 \
  --input '{"files":["orders_a.csv","notes.txt"]}'

The Lambda handler rejects notes.txt.

That failure propagates from the affected Map iteration. The Map’s Catch rule routes the workflow to the Fail state.

The resulting execution should be marked failed rather than silently accepting the invalid input.

Inspect the execution history to understand the failing state, the returned error, and the error transition.

Notice that the Retry policy is intended for temporary Lambda service errors, not invalid input. This is deliberate.

In a real batch-processing platform, you might instead route invalid records into a rejected-records destination and let valid records continue. The correct policy depends on whether the entire batch must succeed atomically.

Step 14: Clean up AWS resources

When you have finished, delete the deployed application:

sam delete \
  --stack-name retail-map-demo \
  --region us-east-1

Follow the CLI prompts and verify that any separately retained resources have been removed.

Deleting the CloudFormation stack avoids leaving unused Lambda functions, state machines, and related infrastructure deployed indefinitely.

What does the AWS lab cost?

There is no AWS service charge for running the Python tests or JSON validation entirely on your own laptop.

Once you call AWS APIs or deploy resources, costs depend on the service and activity.

For Standard workflows in US East (N. Virginia), AWS lists $0.000025 per billable state transition and a free tier of 4,000 state transitions per month.

A small three-file experiment should generally produce negligible usage charges. That does not mean a production workload with millions of executions is automatically cheap.

Reference: AWS Step Functions pricing.

Scaling the example from three files to millions of records

Our example uses Inline Map because its input is a small JSON array.

Now imagine receiving a manifest describing ten million S3 objects.

Passing all object identifiers through one state-machine input is not viable because of workflow payload size limits.

Distributed Map is designed for this situation.

Instead of supplying a huge array in the execution request, the workflow can reference supported S3 data sources and launch child executions.

A production implementation should define:

• An immutable S3 input manifest.

• A child workflow type appropriate for the work.

• A safe maximum concurrency.

• Batch sizes based on downstream capacity.

• Failure thresholds and redrive behavior.

• A durable output destination.

• Permissions for S3 access and child workflow execution.

• Metrics for successful, failed, and retried items.

The goal is not necessarily to launch 10,000 functions simultaneously.

If the downstream database safely supports only 100 concurrent writes, launching 10,000 child workflows may create throttling, retries, and higher cost without increasing useful throughput.

Set concurrency based on measured downstream capacity, not the maximum number advertised by the orchestrator.

Part 4: Choosing an orchestrator for data engineering and machine learning

There is no universal winner between Airflow and Step Functions.

The right choice depends on what causes the workflow to run, what kind of state it manages, how much independent work it coordinates, and who will operate it.

Data engineering pipelines often care about source partitions, freshness, historical reprocessing, and publication of reliable tables.

ML pipelines also care about dataset and model versions, compute requirements, experiment reproducibility, evaluation, and approval gates.

These distinctions influence which orchestration abstractions provide the most value.

Decision matrix: data engineering

Scheduled batch ELT

Airflow is usually an excellent fit when the platform already runs hundreds of scheduled pipelines.

Daily and hourly scheduling, dependency graphs, retry history, pools, and historical backfills are mature parts of its workflow model.

For example, a daily retail warehouse pipeline may need to ingest orders, join product data, refresh aggregated sales tables, and run data quality checks before publishing reporting datasets.

That workflow benefits from explicit data intervals and controlled reprocessing.

Event-driven ingestion

Step Functions becomes particularly attractive when work is triggered by cloud events.

A file arrives in S3, EventBridge routes an event to a Standard state machine, and the state machine validates the file and launches the appropriate transformation.

There is no need to maintain a separate Airflow scheduler solely to process a relatively small number of events.

However, Airflow’s asset-aware scheduling means the difference is not simply scheduled versus event-driven. Airflow can react to published data, while Step Functions can be started on schedules.

The more meaningful distinction is the surrounding platform and operational model.

CDC processing

Change data capture is not usually a collection of unrelated daily batch jobs.

A CDC source emits a continuous stream of inserts, updates, and deletes.

Kinesis, database streams, SQS, and dedicated stream processors are often better suited to the ingestion portion than a traditional DAG.

Airflow or Dagster can still orchestrate periodic compaction, reconciliation, warehouse materialization, and downstream asset publication.

The stream processor handles individual changes; the orchestrator handles the larger data lifecycle.

Historical backfills

Backfills are one of Airflow’s strongest use cases.

Suppose you discover an error affecting 90 days of sales aggregation.

You need to identify the affected logical dates, rerun transformations, replace outputs safely, and monitor progress.

Airflow provides a natural framework for this.

Step Functions can also handle a backfill, especially using Distributed Map over historical manifests. But you must explicitly define how dates, partitions, manifests, and replay policies are managed.

If historical reprocessing is a regular operational activity, prefer a tool that makes it straightforward.

Decision matrix: machine learning

Feature pipelines: treat data as the primary artifact

Feature engineering pipelines transform raw data into reusable attributes consumed by models.

For a retail recommendation model, features might include purchase frequency, recent product interactions, category preferences, and customer activity counts.

A successful feature pipeline must preserve more than its task status.

It should associate outputs with:

• The source data snapshot.

• The transformation version.

• The time range represented by each feature.

• Data quality results.

• Publication time.

• The destination feature store or table.

Asset-oriented orchestration is valuable because feature tables and partitions are the important outputs.

Airflow Assets and Dagster’s software-defined assets naturally express these dependencies.

Training pipelines: orchestrate compute rather than replacing it

Training a large model can require GPUs, distributed workers, checkpoint management, and long-running jobs.

The orchestration system should coordinate that compute, not pretend to replace it.

A typical training workflow may involve:

1. Build a training dataset from a frozen source snapshot.

2. Validate its schema and quality.

3. Submit the training job.

4. Track its completion.

5. Evaluate the resulting model.

6. Register the model artifact.

7. Require approval or deploy according to policy.

SageMaker Pipelines is a natural option when these steps already use SageMaker AI.

Flyte or Kubeflow Pipelines may be preferable for Kubernetes-based infrastructure.

Airflow can coordinate the same lifecycle as part of a broader data platform.

Step Functions can invoke supported managed services directly and wait for their completion.

No ordinary Lambda function needs to sit idle for hours while a separate training job runs.

Hyperparameter sweeps: control the number of experiments

Hyperparameter tuning often involves running many independent training trials.

For example, a retailer might experiment with 200 configurations of a demand forecasting model.

The training service performs the numerical computation. The orchestrator coordinates experiment execution, dependencies, and results.

The design must account for GPU availability, budget, retry behavior, and early stopping.

If the selected training service already provides hyperparameter tuning and scheduling, use those facilities rather than recreating them as thousands of bespoke orchestration tasks.

An orchestration layer should add coordination, not duplicate a mature compute scheduler.

Batch inference: an ideal fan-out workload

Suppose a trained recommendation model must score 20 million product-interaction records.

A reasonable architecture is to partition the inputs, record a manifest, submit independent inference jobs, monitor completion, and publish the results after validation.

Step Functions Distributed Map can work well when inputs are in S3 and inference uses AWS services.

Airflow is equally reasonable when batch inference is part of a daily warehouse or feature pipeline.

The key question is whether inference is primarily an event-driven AWS operation or one stage of a broader scheduled data product.

Real-time inference is not the same as workflow orchestration

A customer viewing product recommendations expects a low-latency response.

Adding several state-machine transitions to every inference request may increase complexity and latency without offering meaningful benefits.

A direct model-serving endpoint, Lambda function, or purpose-built serving infrastructure is generally a better request path.

Step Functions or durable functions may still orchestrate surrounding asynchronous operations, such as collecting feedback, initiating enrichment, or approving a workflow.

Keep latency-sensitive serving separate from long-running orchestration unless there is a compelling reason to combine them.

Retraining after model drift

Model drift monitoring is an event-producing process.

A monitor identifies a potential degradation and emits an event that may initiate retraining.

A robust workflow should not deploy a new model simply because drift crossed a threshold.

The retraining process should validate available data, train candidates, compare metrics with the current model, document results, and apply explicit deployment and rollback policies.

Step Functions Standard is a good choice when those steps use AWS services and require approval gates.

An existing ML orchestration platform may be preferable when retraining is integrated with experiment tracking, feature lineage, and model governance.

Part 5: Cost, duration, testing, observability, and vendor lock-in

A decision matrix provides a starting point. Production architecture requires a more detailed comparison.

Rule 1: Calculate the complete cost of the workload

The cost of orchestration is not the same as the cost of computation.

For Airflow, include persistent infrastructure, workers, the metadata database, logging, remote compute, and engineering operations.

For Step Functions, include state transitions or Express execution charges, Lambda, event services, logging, storage, and the compute services invoked by the workflow.

A useful comparison is:

Airflow total cost = orchestration infrastructure + worker capacity + metadata services + monitoring + operational effort + external compute.

AWS serverless total cost = Step Functions + Lambda + events + storage + logging + external compute + operational effort.

Both can be economical.

For a dozen brief workflows per month, a fully managed usage-based option may avoid substantial fixed infrastructure costs.

For continuous, high-throughput pipelines, the balance becomes more workload-specific.

A million small tasks can become expensive in any orchestrator if each one carries unnecessary control-plane overhead.

Batch work at a meaningful unit of processing.

Rule 2: The orchestrator’s duration is not the worker’s duration

A Standard Step Functions workflow can coordinate work for up to one year.

That does not mean a standard Lambda invocation can execute for one year.

Similarly, Airflow can coordinate a six-hour Spark job without requiring the Python task itself to perform six hours of Spark computation.

Separate the duration of the logical workflow from the lifetime of its compute jobs.

For long-running Spark, batch, or training jobs, use appropriate remote compute and supported job-completion integrations.

For long-lived application waits and approvals, consider Standard Step Functions, Temporal, or durable Lambda functions.

Rule 3: Observability must include data outcomes

A green task status is not enough.

A pipeline may succeed technically while processing zero records, publishing late data, or using an incorrect source snapshot.

Production monitoring should capture:

• Total workflow duration.

• Input data freshness.

• Number of records processed and rejected.

• Retry and failure rates.

• Data quality results.

• Output partition or artifact identifiers.

• Cost per completed workload.

• Downstream publication status.

Airflow provides detailed DAG and task history, together with asset-oriented metadata.

Step Functions provides execution histories, visual state transitions, and Map Run monitoring.

Both benefit from explicit application metrics, lineage, and data quality checks.

Rule 4: Design local testing into the code

The most productive workflow implementations separate domain logic from orchestration code.

A Python validation function should be testable without initializing an entire Airflow instance or creating an AWS account.

Airflow’s dag.test() is useful for executing actual DAG logic locally, but that is distinct from unit-testing an individual transformation.

Likewise, pytest can validate a Lambda handler without AWS credentials, but it cannot prove that the IAM role or state-machine integration is correctly deployed.

A strong testing strategy combines:

• Unit tests for transformation logic.

• DAG or workflow structural checks.

• Local execution where supported.

• Cloud-side definition validation for AWS ASL.

• Integration tests in an isolated environment.

• Failure and replay tests before production deployment.

The distinction between these layers prevents false confidence.

Rule 5: Reduce vendor lock-in by isolating business logic

Airflow is open source, but Airflow DAGs using AWS-specific operators still depend on AWS.

Step Functions is more directly tied to the AWS ecosystem because ASL integrations and execution semantics are provided by AWS.

The most practical portability strategy is not attempting to make every workflow definition universal.

Instead, keep transformation code behind clear contracts.

A task that reads a documented input artifact and produces a documented output artifact can be invoked by Airflow, AWS Batch, a container service, or another orchestrator.

That design makes future migrations considerably easier.

Part 6: Cherry on the cake — how Capital One achieved up to 80% faster processing with Distributed Map

A particularly useful real-world case comes from Capital One and its AWS Step Functions-based check-clearing application.

The team was already using Step Functions and Lambda.

Its existing workflow relied on Inline Map, which limited concurrent iterations to 40.

Capital One adopted Distributed Map to increase the number of workflows that could launch and close concurrently.

According to the AWS customer case study, the change delivered the following results:

Reported workload-specific results from the AWS Capital One customer case study; not general performance guarantees.

Capital One reported a 75–80% reduction in overall workflow launch and closure processing time and the ability to launch 25 times as many concurrent workflows as the previous batches of 40.

The implementation did not require a wholesale rewrite of the application architecture.

The performance improvement came largely from changing the orchestration fan-out model rather than rewriting the business processing code.

Read the case: Processing Checks Up to 80% Faster Using AWS Step Functions Distributed Map with Capital One.

Why this story matters

The lesson is broader than the specific AWS feature.

When a pipeline is slow, it is tempting to add CPU, increase memory, or optimize individual functions.

But the bottleneck may be the orchestration model itself.

A workflow limited to launching 40 independent units at a time may underutilize a downstream platform capable of processing substantially more.

Changing the fan-out strategy can then provide a major improvement without touching the underlying business logic.

The opposite is also true.

If the actual bottleneck is a database that supports only 50 safe concurrent writes, increasing orchestration concurrency to 10,000 may worsen performance.

Before changing orchestrators, identify whether the bottleneck is:

• Scheduling and orchestration overhead.

• Worker startup time.

• Actual computation.

• Network or storage throughput.

• API rate limits.

• Downstream service concurrency.

• Excessive retries.

• Poor batching strategy.

Capital One’s results are compelling because they show how much workflow architecture can matter. They do not imply that Distributed Map automatically delivers the same improvement for every application.

Part 7: A practical orchestration selection checklist

Before choosing a workflow system, answer five questions.

1. What is the fundamental unit of work?

If the important object is a warehouse table, feature partition, or published dataset, an asset-oriented orchestrator is a strong candidate.

If the important object is a long-lived business operation with recoverable steps, consider Step Functions, Temporal, or durable functions.

If the primary object is an ML experiment or training artifact, a specialized ML platform may provide greater value.

2. What triggers execution?

Scheduled data intervals often suit Airflow.

Object arrivals, application events, and AWS service notifications often suit EventBridge and Step Functions.

Data asset updates can naturally trigger Airflow or Dagster workflows.

The trigger is useful evidence, but it should not be the sole deciding factor.

3. How much independent work exists?

Identify the actual safe concurrency limit.

It may be governed by GPU availability, database capacity, vendor API quotas, warehouse concurrency, or available worker memory.

Use Airflow pools, Step Functions Map limits, or compute-platform controls to enforce those limits.

Unlimited orchestration concurrency is rarely a useful production objective.

4. How will failed work be replayed?

Choose a replay unit: event, record, file, partition, job, or entire workflow.

Store immutable input references and stable execution identifiers wherever possible.

Define whether reruns replace, merge, skip, or version previous output.

Verify the policy by deliberately causing failures in a test environment.

5. Who will operate the platform?

An Airflow deployment requires people who understand DAG authoring, executors, scheduler behavior, upgrades, metadata storage, and provider compatibility.

Step Functions requires engineers who understand IAM, AWS quotas, state-machine errors, service integrations, and usage costs.

ML orchestration requires additional expertise in datasets, model artifacts, experiment tracking, and training infrastructure.

The best architecture is one the team can reliably debug and maintain.

Final recommendations

Choose Airflow 3.x for broad data engineering orchestration with strong scheduling, historical replay, heterogeneous systems, Python authoring, and increasingly capable asset dependencies.

Choose Dagster when software-defined data assets, lineage, quality checks, and partitions are the central abstractions.

Choose AWS Step Functions for AWS-native event-driven workflows that benefit from managed execution, explicit states, direct service integrations, and limited control-plane maintenance.

Choose Distributed Map when large-scale, independent fan-out is fundamental to the workload, especially when the source data already resides in S3.

Choose Lambda durable functions for code-centric, recoverable application workflows involving checkpoints, delays, and long-lived logical processes.

Choose Lambda Managed Instances when sustained utilization, specialized instance capacity, or longer supported invocations justify EC2-based economics.

For ML, choose the execution environment and orchestration platform together. SageMaker Pipelines is a strong AWS-native choice, while Flyte and Kubeflow Pipelines suit Kubernetes-centric ML systems. Airflow and Step Functions remain useful coordinators for the wider data and application lifecycle.

Your next step: build, measure, and compare

Start with the local Airflow lab. Confirm that three runtime-discovered filenames become three mapped task instances, that the manifest contains five rows, and that the asset update triggers the consumer DAG.

Then run the Lambda pytest suite, validate the ASL structure, and—if you have a suitable AWS sandbox account—deploy the Step Functions workflow.

Change the number of files. Introduce a deliberately invalid filename. Inspect the execution histories, logs, retries, and failure transitions.

Then compare the approaches using four measurements: time to implement, time to recover from failure, cost per completed workload, and operational effort.

Finally, select one small, non-sensitive workflow from your own environment and prototype it using two orchestration approaches before committing to a production platform.

The objective is not to find one orchestrator that wins every comparison. It is to choose the system whose execution, failure, concurrency, and data dependency model best matches the workload you actually need to operate.