The data and ML tool map: hello worlds for dbt, Airflow, Spark, MLflow and friends, and when to use each

If you are new to data engineering or machine learning, the modern tool landscape can feel unnecessarily complicated.

One tutorial tells you to learn Apache Spark. Another insists that SQL and dbt are all you need. A third introduces Kafka, Airflow, a feature store, a model registry, and Kubernetes before you have even processed your first CSV file.

The confusing part is that these tools are not necessarily competitors. Most solve different problems.

The most useful question is not “Which tool is best?” It is “What problem does this tool solve, and do I have that problem yet?”

This guide builds a practical map of the ecosystem, from moving and transforming data to training, deploying, and monitoring machine learning models.

You will learn:

• The vocabulary data engineers and ML engineers use every day.

• Six common architecture patterns and why teams choose them.

• Local, runnable hello worlds for the most important tools.

• How to distinguish tools that complement one another from tools that overlap.

• When a simple Python script is enough, and when distributed infrastructure becomes worthwhile.

Our running example is an imaginary online shop. It records customer orders, produces sales reports, and eventually trains a model to estimate delivery times.

All examples use Python 3.12, contemporary package releases available in October 2026, and a single laptop. No cloud account or API key is required. Docker is optional except for the local Kafka demonstration, where we use it to run a broker.

Part 1: The vocabulary — learn the concepts before the products

Tools change faster than the fundamental problems they solve. Once you understand the following concepts, most data engineering architecture diagrams become much easier to read.

1. Data movement and transformation

Source

A source is where data originates or where a pipeline reads it. It might be an application database, an API, a CSV file, a message topic, or another dataset. For our online shop, a PostgreSQL orders table is a source. One pipeline can have several sources, such as orders, product information, and website events.

Sink (data sink)

A sink is a destination to which a pipeline writes data. Examples include a warehouse table, a Parquet file, a search index, or a Kafka topic. If our shop extracts orders from PostgreSQL and writes them into DuckDB, the DuckDB table is the sink for that operation. A sink for one pipeline may become a source for the next.

Ingestion

Ingestion is the process of bringing data into a system, usually from an external source. It might involve downloading a file every morning, copying database rows, or continuously consuming events. For example, an ingestion script retrieves yesterday’s orders and saves them to object storage. Ingestion does not necessarily include business transformations.

ETL versus ELT

Both approaches move and transform data, but the order differs.

• ETL — Extract, Transform, Load: Read data, clean or reshape it in a processing system, and then load the result into its destination.

• ELT — Extract, Load, Transform: Read data, load it into the destination, and perform transformations there.

With ETL, Python might clean order records before inserting them into a warehouse. With ELT, raw orders arrive in the warehouse first, and dbt runs SQL transformations afterward.

ETL is useful when data must be transformed before reaching the destination. ELT is especially attractive when the destination already has a capable analytical engine.

Batch versus streaming

Batch processing works on bounded collections of data, often on a schedule. Streaming processes data incrementally as new records arrive, potentially without a defined endpoint.

For example, a batch job calculates yesterday’s revenue at 2 AM. A streaming job updates a live sales counter as orders occur.

Streaming is not automatically better. A daily report rarely benefits from millisecond-level processing, while a real-time anomaly alert may genuinely require it.

2. Pipelines and reliability

Pipeline and DAG

A data pipeline is a sequence of operations that produce a desired output. A DAG, or directed acyclic graph, represents tasks and their dependencies without dependency cycles.

For example, a revenue report might require extracting orders, validating them, aggregating totals, and publishing results. The aggregation cannot execute successfully before its required input is ready.

A DAG specifies dependency relationships; it does not necessarily require tasks to execute one at a time.

Orchestrator

An orchestrator manages workflow execution: deciding when tasks run, checking dependencies, recording failures, retrying jobs, and showing operational status.

Airflow and Prefect are examples.

An orchestrator typically does not perform large transformations itself. It starts and coordinates systems that do.

For our store, an orchestrator could start a daily ingestion script, wait for it to finish, run a dbt transformation, and notify the team if the final report fails.

Idempotency

An operation is idempotent if repeating it produces the same intended final state as running it once.

Imagine a daily revenue job that fails halfway through. If rerunning it inserts another copy of yesterday’s data, the process is not idempotent.

A better design replaces the target date’s partition or upserts records using stable order IDs. Retries then become much safer.

Idempotency matters because production pipelines inevitably encounter interruptions and retries.

Backfill

A backfill processes historical data that was missed or needs to be recomputed.

Suppose a revenue calculation contained a bug from September 1 through September 10. After fixing the transformation, you rerun those ten days.

That is a backfill.

Reliable pipelines accept an explicit processing date or date range instead of assuming every execution should process the current day.

CDC — Change Data Capture

CDC identifies changes to source data and makes them available downstream.

Instead of repeatedly copying every row from an orders database, a CDC system can capture inserts, updates, and deletes from a database’s change log.

For example, when an order changes from pending to shipped, a downstream system receives a corresponding change event.

CDC supports efficient synchronization, but consumers must still handle duplicated messages, ordering, and deletes correctly.

3. Data representation and storage

Schema, schema-on-read, and schema-on-write

A schema defines the structure of data: column names, types, nullability, and sometimes constraints.

An orders schema might specify an integer order ID, a timestamp, a customer ID, and a decimal total.

Schema-on-write validates or enforces that structure when data is stored. Schema-on-read interprets or applies structure when data is accessed.

A relational database commonly enforces a declared schema on writes. A raw JSON file collection may defer interpretation until queries read it.

These approaches are not mutually exclusive across an entire platform.

Data warehouse versus data lake versus lakehouse

A data warehouse is designed for structured analytics, SQL queries, reporting, and governed datasets. A retailer might store curated sales tables in Snowflake or BigQuery.

A data lake stores files such as Parquet, JSON, images, and logs, commonly in object storage. It provides flexible storage, but file storage alone does not give you transactional tables or reliable concurrent updates.

A lakehouse combines lake-style storage with table-management capabilities such as transactional updates, metadata, and consistent snapshots.

A lakehouse is an architectural approach, not simply a different name for a data lake.

Table formats: Apache Iceberg and Delta Lake

Parquet is a columnar file format. Iceberg and Delta Lake are table formats that organize data files into reliable logical tables.

They provide capabilities such as snapshot-based reads, schema evolution, transactional commits, and controlled table updates, with the exact behavior depending on the format and engine.

For example, an order table might consist of hundreds of Parquet files while Iceberg or Delta metadata identifies which files belong to its current snapshot.

You do not need either format merely to query Parquet files on a laptop.

Partitioning

Partitioning divides a dataset into logical subsets based on values such as dates or regions.

A sales dataset might be partitioned by order date. Queries requesting one particular day can potentially avoid scanning unrelated partitions.

Good partitioning reduces unnecessary work. Excessively fine partitioning, however, can create thousands of tiny files and costly metadata operations.

Partition pruning also depends on how data is stored and how queries are written.

4. Analytics modeling

Staging and marts

Staging models are early transformations that standardize raw data. They might rename columns, convert timestamps, normalize status values, and remove invalid records.

Data marts contain business-oriented datasets designed for a particular analytical purpose.

For example, a staging model standardizes order totals, while a sales mart calculates daily revenue by product category.

A useful distinction is that staging models speak the language of the source; marts increasingly speak the language of the business.

Bronze, silver, and gold — the medallion layers

The medallion pattern organizes data into progressively refined layers:

• Bronze: Raw or minimally modified incoming data.

• Silver: Cleaned, validated, deduplicated, and standardized data.

• Gold: Business-ready aggregates, metrics, and analytical models.

An online order event might enter bronze as JSON, become a validated order row in silver, and contribute to a daily revenue table in gold.

These are logical layers. They do not require three separate databases or three separate compute clusters.

Fact versus dimension tables — the star schema

A fact table records measurable business events, usually at a clearly defined grain. A dimension table describes entities involved in those events.

For a retailer:

• A fact table might contain one row per order line, with quantity and revenue.

• A product dimension might describe the product name, category, and brand.

• A date dimension might describe calendar dates, weeks, months, and fiscal periods.

A star schema places facts at the center and connects them to dimensions using keys.

The important design decision is the grain. Mixing order-level and order-line-level measurements without care creates double-counting bugs.

Slowly changing dimensions (SCDs)

A slowly changing dimension handles changes in descriptive attributes over time.

Suppose a product moves from one category to another.

With a Type 1 approach, the existing category is overwritten. Historical reports then use the current category.

With a Type 2 approach, a new version of the product record is added with effective dates. Historical reports can preserve the category that was valid when the order occurred.

Neither approach is universally better; the business meaning determines the choice.

5. Trust, governance, and quality

Data contract

A data contract is an agreement between data producers and consumers about expected structure and behavior.

For example, the checkout service may promise that every order event includes an order ID, timestamp, currency, and valid status.

A stronger contract also documents ownership, compatibility rules, and expectations about delivery and freshness.

Contracts reduce surprise breakages when upstream software changes.

Data lineage

Lineage describes where a dataset came from and which transformations contributed to it.

Suppose the daily revenue dashboard looks incorrect. Lineage can help identify the mart, its staging models, and the upstream orders source.

Column-level lineage goes further by tracing individual output columns back through transformations.

Lineage is especially valuable during debugging, compliance reviews, and schema changes.

Data quality checks

Data quality checks assert properties that data should satisfy.

For example:

• Order IDs must not be null.

• Revenue must be nonnegative.

• Order IDs must be unique if the table’s grain is one row per order.

• Every order must reference a known customer.

• Yesterday’s data must arrive before a defined deadline.

Quality includes validity, completeness, uniqueness, consistency, and freshness.

Passing checks increases confidence, but does not prove that every business calculation is correct.

6. Machine learning infrastructure

Feature store

A feature store manages reusable, defined inputs to machine learning models.

For example, a shop’s delivery model might use a customer’s recent order count or a warehouse’s average dispatch delay.

A feature store helps manage feature definitions, historical retrieval, and sometimes low-latency serving.

Not every ML project needs one. A small project with a single training pipeline can often compute features directly.

Offline versus online features

Offline features are generally retrieved for training, backtesting, and batch predictions. Online features are made available quickly for individual prediction requests.

For example, historical customer order counts might be read from Parquet during training, while the latest counts are fetched from a low-latency online store during checkout.

The difficult part is consistency: a model should see features with compatible definitions during training and inference.

Historical feature retrieval also needs to respect what information was available at the prediction time, otherwise data leakage can occur.

Experiment tracking

Experiment tracking records what happened during model development: parameters, metrics, model artifacts, code versions, and other useful context.

For example, you train three delivery-time models with different configurations. Tracking tells you which run achieved the lowest validation error and which configuration produced it.

MLflow is a common tool for this.

Model registry

A model registry manages model identities, versions, metadata, and lifecycle information.

For example, a team might maintain a registered delivery model with multiple versions and an alias identifying the version currently approved for production use.

Experiment tracking answers, “What did we try?” A registry helps answer, “Which model version are we managing or deploying?”

The registry does not automatically deploy a model simply because it has been registered.

Training versus inference and serving

Training fits model parameters using examples. Inference applies a trained model to new input data.

Serving makes inference accessible to consumers, often through an HTTP API or a batch processing system.

For example, training learns the relationship between delivery distance and delivery time. Inference estimates a time for a new order.

Training is not serving. A model trained successfully in a notebook is not automatically available to your application.

Part 2: Six architecture patterns you will encounter

Now we can connect the concepts into recognizable system designs.

The diagrams are intentionally small. Real deployments add authentication, logging, monitoring, access controls, failure handling, and many other details.

Pattern 1: Classic batch ETL into a warehouse

+------------------+
| Source databases |
| APIs and files   |
+--------+---------+
         |
         v
+------------------+
| Extract          |
+--------+---------+
         |
         v
+------------------+
| Transform        |
| Python / Spark   |
+--------+---------+
         |
         v
+------------------+
| Data warehouse   |
+--------+---------+
         |
         v
+------------------+
| BI and reports   |
+------------------+

When to choose it: Your data arrives periodically, substantial preprocessing must happen before loading, or the destination has limited transformation capabilities.

Imagine receiving supplier CSV files with inconsistent product codes. A Python job validates the files, standardizes codes, and loads accepted rows into a warehouse.

The architecture is straightforward and can be implemented with a script plus a scheduler.

The important operational decisions are how to handle invalid records, incremental loads, retries, and historical reruns.

You do not need Spark just because the diagram contains a transformation step.

Pattern 2: ELT with dbt on a warehouse

+------------------+
| Apps / databases |
+--------+---------+
         |
         v
+------------------+
| Ingestion / CDC  |
+--------+---------+
         |
         v
+------------------+
| Warehouse        |
| Raw tables       |
+--------+---------+
         |
         v
+------------------+
| dbt SQL models   |
| Staging -> Marts |
| Tests and docs   |
+--------+---------+
         |
         v
+------------------+
| Dashboards       |
+------------------+

When to choose it: Your organization already uses a capable analytical database, transformations are primarily SQL, and you want reusable, version-controlled models.

This architecture separates ingestion from business transformation.

The ingestion system places raw orders in the warehouse. dbt then expresses transformations as SQL models and manages their dependencies.

A staging model standardizes orders. A mart calculates revenue. Tests check important assumptions.

dbt does not replace the database’s query engine. Its SQL runs in the database through an adapter.

It also does not automatically replace an operational scheduler or ingestion tool.

For teams comfortable with SQL, this is often an excellent starting point.

Pattern 3: Lambda versus kappa — two approaches to streaming

Lambda architecture maintains separate batch and streaming processing paths.

                 +----------------+
                 | Event source   |
                 +-------+--------+
                         |
              +----------+----------+
              |                     |
              v                     v
      +---------------+     +---------------+
      | Batch layer   |     | Speed layer   |
      | Historical    |     | Low latency   |
      +-------+-------+     +-------+-------+
              |                     |
              +----------+----------+
                         |
                         v
                 +---------------+
                 | Serving view  |
                 +---------------+

Its main attraction is that batch processing can recompute historical results while a speed layer provides fresh updates.

Its biggest disadvantage is maintaining two processing paths that should calculate the same business result.

Kappa architecture aims to use one streaming-oriented processing path, including replaying historical events when necessary.

+------------------+
| Event producers  |
+--------+---------+
         |
         v
+------------------+
| Retained event   |
| log / Kafka      |
+--------+---------+
         |
         v
+------------------+
| Stream processor |
+--------+---------+
         |
         v
+------------------+
| Materialized     |
| views / sinks    |
+------------------+

When to choose lambda: You have genuinely different batch and low-latency requirements and are prepared to maintain both paths.

When to choose kappa: Your processing logic can operate over an event stream, replay is practical, and you want to avoid separate batch and streaming implementations.

Neither architecture eliminates complexity.

Kafka retains and distributes events, but it does not automatically calculate your business aggregates. You still need processing and state-management logic.

Likewise, a replayable log does not guarantee infinite retention. Historical reprocessing depends on what events remain available or have been archived.

For a small online shop, start with batch reports unless a concrete requirement makes streaming necessary.

Pattern 4: Lakehouse with medallion layers

+----------------------+
| Databases / events   |
+----------+-----------+
           |
           v
+----------------------+
| Bronze               |
| Raw files / records  |
+----------+-----------+
           |
           v
+----------------------+
| Silver               |
| Validated tables     |
+----------+-----------+
           |
           v
+----------------------+
| Gold                 |
| Business datasets    |
+----------+-----------+
           |
           v
+----------------------+
| Analytics / ML       |
+----------------------+

Storage: Parquet + table metadata
Engine: Spark / another query engine

When to choose it: You want flexible, file-based analytical storage, multiple processing engines, and robust management of large datasets.

A retailer might preserve raw checkout events, create a clean orders table, and build revenue marts without copying every layer into a traditional warehouse.

Iceberg or Delta Lake can manage logical tables over the data files.

The processing engine and table format are distinct choices. Spark can work with lakehouse tables, but the lakehouse architecture does not mean that every operation must run on Spark.

A lakehouse is not a free shortcut around governance. You still need schemas, tests, ownership, access controls, and maintenance.

Pattern 5: Event-driven pipeline with a message queue

+------------------+
| Checkout app     |
+--------+---------+
         |
         v
+------------------+
| Message broker   |
| Kafka topic      |
+--------+---------+
         |
         +--------------------+
         |                    |
         v                    v
+----------------+   +----------------+
| Consumer A     |   | Consumer B     |
| Update metrics |   | Send receipt   |
+--------+-------+   +--------+-------+
         |                    |
         v                    v
+----------------+   +----------------+
| Analytics sink |   | Email service  |
+----------------+   +----------------+

When to choose it: Independent systems need to react to events without becoming tightly coupled, or events must be buffered and processed asynchronously.

For example, checkout publishes an order-created event. Analytics and notification services consume it independently.

Kafka is a distributed event-streaming platform with durable, partitioned logs. A traditional work queue often has different consumption and retention semantics.

That distinction matters: Kafka consumers track positions in a log; records are not normally deleted simply because one consumer processed them.

Consumers must also be designed for failures and repeated delivery. An emailed receipt, for instance, should not be sent twice merely because a message was replayed.

Pattern 6: The machine learning lifecycle loop

+------------------+
| Data sources     |
+--------+---------+
         |
         v
+------------------+
| Data validation  |
+--------+---------+
         |
         v
+------------------+
| Feature creation |
+--------+---------+
         |
         v
+------------------+
| Model training   |
+--------+---------+
         |
         v
+------------------+
| Experiment       |
| tracking         |
+--------+---------+
         |
         v
+------------------+
| Model registry   |
+--------+---------+
         |
         v
+------------------+
| Serving / batch  |
| inference        |
+--------+---------+
         |
         v
+------------------+
| Monitoring       |
+--------+---------+
         |
         +-------> New training cycle

When to choose it: A machine learning model is becoming a maintained product rather than a one-off experiment.

Our shop might start with a delivery-time prediction model.

Initially, a single Python script trains and evaluates it. As the application evolves, we may add experiment tracking, model registration, scheduled retraining, API serving, and monitoring.

Monitoring should cover more than server uptime. Useful signals include input distribution changes, prediction latency, errors, and model quality when delayed ground-truth outcomes become available.

Crucially, retraining should not automatically promote every newly trained model to production. Promotion should follow evaluation and release criteria.

The complete diagram represents responsibilities, not a mandatory shopping list.

Part 3: The tools — runnable hello worlds on one laptop

We will now implement small examples of the tools that appear most frequently in modern data and ML stacks.

These are deliberately independent exercises, not a giant production application. Each example introduces one responsibility before adding operational complexity.

0. Set up the environment

Use Python 3.12 on Linux, macOS, or Windows through WSL2.

The shell commands assume a POSIX-compatible terminal. On native Windows, virtual environment activation and some commands differ.

Create a project directory and a virtual environment:

mkdir data-tool-map
cd data-tool-map

python3.12 -m venv .venv
. .venv/bin/activate

python -m pip install --upgrade pip

Install the tools used by the smaller examples:

python -m pip install \
  "pandas==3.0.6" \
  "polars==2.0.0" \
  "duckdb==1.5.6" \
  "scikit-learn==1.9.1" \
  "pandera[pandas]==0.34.1" \
  "mlflow==3.17.1" \
  "fastapi==0.143.2" \
  "uvicorn==0.54.0" \
  "confluent-kafka==2.16.0"

These are pinned releases available as of October 2026, not floating installations of whatever happens to be newest next year. For example, the pinned versions reflect the contemporary Polars 2.x, scikit-learn 1.9, and MLflow 3.x APIs.

Airflow, dbt, Prefect, and Spark receive their own virtual environments later. This is intentional: large frameworks can have different dependency requirements, and isolation avoids many unnecessary conflicts.

Docker requirements: pandas, Polars, DuckDB, dbt, Airflow, Prefect, local Spark, Pandera, scikit-learn, MLflow, and FastAPI do not require Docker for these exercises. Only the Kafka example uses Docker.

1. pandas: the classic Python DataFrame

What it is: pandas is a Python library for manipulating tabular data in memory. Its DataFrame abstraction provides familiar operations such as filtering, grouping, joining, and aggregating.

Use it when: You are exploring data, preparing datasets, writing straightforward transformations, or working with the broader Python scientific-computing ecosystem.

Do not reach for it first when: Your workload is larger than available memory, SQL is the natural interface, or you need a distributed processing framework.

Create pandas_hello.py:

import pandas as pd

orders = pd.DataFrame(
    {
        "order_id": [101, 102, 103, 104],
        "category": ["books", "electronics", "books", "books"],
        "amount": [25.0, 120.0, 15.0, 30.0],
    }
)

sales = (
    orders.groupby("category", as_index=False)
    .agg(
        revenue=("amount", "sum"),
        order_count=("order_id", "count"),
    )
    .sort_values("category")
)

print(sales)

Run:

python pandas_hello.py

Expected result:

      category  revenue  order_count
0        books     70.0            3
1  electronics    120.0            1

The important operation is grouping rows by category and calculating two different aggregates.

This pattern appears everywhere in analytics. Later, DuckDB and Spark will perform essentially the same operation through different execution engines.

2. Polars: fast, expression-based DataFrames

What it is: Polars is an analytical DataFrame engine implemented primarily in Rust. It supports eager and lazy execution, parallel processing, and query optimization.

Use it when: You prefer Python expressions, need efficient transformations, or want to process large analytical datasets on a single machine.

Do not reach for it first when: Your workload depends heavily on pandas-specific libraries or you need an established distributed compute environment.

Create polars_hello.py:

import polars as pl

orders = pl.DataFrame(
    {
        "order_id": [101, 102, 103, 104],
        "category": ["books", "electronics", "books", "books"],
        "amount": [25.0, 120.0, 15.0, 30.0],
    }
)

sales = (
    orders.lazy()
    .group_by("category")
    .agg(
        pl.col("amount").sum().alias("revenue"),
        pl.len().alias("order_count"),
    )
    .sort("category")
    .collect()
)

print(sales)

Run:

python polars_hello.py

Notice the lazy execution pattern.

Rather than executing every operation immediately, Polars builds a query plan and executes it when collect() is called. That allows the engine to optimize the computation.

For larger file-based workloads, Polars also supports lazy scans and streaming execution.

pandas versus Polars: pandas is extremely flexible and deeply integrated into the Python ecosystem. Polars emphasizes optimized analytical expressions and efficient execution.

You can learn both without treating either as a universal replacement for the other.

3. DuckDB: analytical SQL without a database server

What it is: DuckDB is an embedded analytical database. You can execute SQL from Python without deploying a separate server.

It supports files such as CSV and Parquet, persistent database files, analytical SQL, joins, and aggregations.

Use it when: You want SQL analytics locally, need to query Parquet files, or want a lightweight database for analytical transformations.

Do not reach for it first when: You need a conventional transactional application database with many independent concurrent writers, or processing genuinely requires several machines.

Create duckdb_hello.py:

import duckdb

con = duckdb.connect("retail.duckdb")

con.execute(
    """
    CREATE OR REPLACE TABLE orders AS
    SELECT *
    FROM (
        VALUES
            (101, 'books', 25.0),
            (102, 'electronics', 120.0),
            (103, 'books', 15.0),
            (104, 'books', 30.0)
    ) AS t(order_id, category, amount)
    """
)

result = con.execute(
    """
    SELECT
        category,
        SUM(amount) AS revenue,
        COUNT(*) AS order_count
    FROM orders
    GROUP BY category
    ORDER BY category
    """
).fetchall()

for row in result:
    print(row)

con.close()

Run:

python duckdb_hello.py

You now have a persistent database file named retail.duckdb.

The entire example uses normal SQL. There is no database daemon, administrator account, or cloud connection.

A particularly useful DuckDB capability is querying Parquet directly rather than importing every file into a traditional database table.

A useful mental model:

• pandas and Polars are DataFrame-oriented analytical tools.

• DuckDB is a SQL-oriented analytical engine.

• All three can work on one machine.

• Spark becomes attractive when distributed execution is genuinely needed.

4. dbt: SQL transformations as a maintainable project

What it is: dbt organizes SQL transformations into dependency-aware models with tests, documentation, configurations, and generated artifacts.

dbt Core is the command-line transformation framework. An adapter connects it to the SQL engine that performs the actual work.

Here we use DuckDB as that engine, so no cloud warehouse is necessary.

Use it when: Multiple SQL transformations must be maintained together, reviewed in version control, tested, and reused by analysts.

Do not reach for it first when: You only have one trivial SQL statement, need arbitrary event processing, or expect dbt alone to ingest data from operational systems.

Install dbt Core and the DuckDB adapter

Create a separate environment from the project root:

deactivate

python3.12 -m venv .dbt-venv
. .dbt-venv/bin/activate

python -m pip install \
  "dbt-core==1.12.6" \
  "dbt-duckdb==1.11.0" \
  "duckdb==1.5.6"

The adapter supports local DuckDB connections, while dbt Core provides the project runner.

Create the project

Build this directory structure:

shop_dbt/
├── dbt_project.yml
├── profiles.yml
├── seeds/
│   └── raw_orders.csv
└── models/
    ├── sources.yml
    ├── stg_orders.sql
    ├── mart_daily_sales.sql
    └── schema.yml

Here is what each file does.

File: dbt_project.yml

name: shop_demo
version: "1.0.0"
config-version: 2

profile: shop_demo

model-paths: ["models"]
seed-paths: ["seeds"]

models:
  shop_demo:
    +materialized: table

This declares the dbt project and configures its models to materialize as database tables.

File: profiles.yml

shop_demo:
  target: dev
  outputs:
    dev:
      type: duckdb
      path: shop.duckdb
      schema: main
      threads: 1

The profile defines the database connection. Our target is a local DuckDB file.

File: seeds/raw_orders.csv

order_id,order_date,category,amount
101,2026-10-01,books,25.00
102,2026-10-01,electronics,120.00
103,2026-10-02,books,15.00
104,2026-10-02,books,30.00

Seeds allow dbt to load small CSV files into the database.

In a production ELT pipeline, a separate ingestion system would usually populate raw operational data. We use a seed to make the exercise entirely self-contained.

File: models/sources.yml

version: 2

sources:
  - name: shop
    schema: main
    tables:
      - name: raw_orders

This declares a source table. Other models can refer to it by its logical source name.

File: models/stg_orders.sql

SELECT
    CAST(order_id AS INTEGER) AS order_id,
    CAST(order_date AS DATE) AS order_date,
    CAST(category AS VARCHAR) AS category,
    CAST(amount AS DECIMAL(10, 2)) AS amount
FROM {{ source('shop', 'raw_orders') }}
WHERE amount >= 0

This is our staging model.

It standardizes types and rejects rows with negative amounts. The source function is evaluated by dbt’s Jinja templating system.

The staging model is not an independent Python ETL script. dbt compiles the model into SQL that DuckDB executes.

File: models/mart_daily_sales.sql

SELECT
    order_date,
    COUNT(*) AS order_count,
    SUM(amount) AS revenue
FROM {{ ref('stg_orders') }}
GROUP BY order_date

The ref() function declares a dependency on the staging model.

dbt uses this relationship to construct its model DAG and determine the correct execution order.

File: models/schema.yml

version: 2

models:
  - name: stg_orders
    columns:
      - name: order_id
        data_tests:
          - not_null
          - unique

We have added two generic data tests.

The not_null test rejects missing order IDs. The unique test rejects duplicate order IDs.

These tests are appropriate because our example has one row per order. An order-line fact table would require a different uniqueness key.

Run the project

From the shop_dbt directory, with the dbt environment active:

dbt debug --profiles-dir .

dbt seed --profiles-dir .

dbt build --profiles-dir .

The initial seed command is intentional. The staging model declares the raw table as a source, so seeding it first guarantees that the source exists before the models run.

dbt build then builds selected project resources and runs their associated tests. It supports seeds, models, tests, and other supported resource types in the project dependency graph.

To inspect the resulting mart, run:

dbt show --select mart_daily_sales --profiles-dir .

Expected business results:

This tiny project demonstrates the central value of dbt:

• The raw data source is declared explicitly.

• Staging logic is separated from business logic.

• Model dependencies are defined in code.

• Tests run as part of the build.

• Transformations can be reviewed and version-controlled.

As models accumulate, these capabilities become far more valuable than they appear in a two-model tutorial.

5. Apache Airflow: orchestrate dependent tasks

What it is: Apache Airflow is a workflow orchestration platform for defining, scheduling, observing, and managing tasks.

Airflow is particularly useful for recurring batch-oriented workflows with dependencies, operational visibility, and established scheduling requirements.

Use it when: You have multiple dependent jobs, retries, schedules, historical runs, and a team that needs a shared orchestration interface.

Do not reach for it first when: You only need to run one local script occasionally, or you expect the orchestrator itself to replace a distributed processing engine.

Install Airflow 3

Airflow has substantial dependencies, so use another environment.

From the original project directory:

deactivate

python3.12 -m venv .airflow-venv
. .airflow-venv/bin/activate

python -m pip install "apache-airflow==3.3.2" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.3.2/constraints-3.12.txt"

export AIRFLOW_HOME="$(pwd)/airflow_home"
mkdir -p "$AIRFLOW_HOME/dags"

The constraints file matters. Airflow publishes dependency constraints for a specific Airflow and Python version to make installations reproducible.

Airflow 3.3.2 was released in September 2026, and modern DAG authoring uses the public airflow.sdk interface.

Create a minimal TaskFlow DAG

Save the following as airflow_home/dags/retail_hello.py:

from airflow.sdk import dag, task


@dag(
    dag_id="retail_hello",
    schedule=None,
    catchup=False,
    tags=["tutorial"],
)
def retail_hello():
    @task
    def count_orders() -> int:
        orders = [101, 102, 103]
        return len(orders)

    @task
    def print_report(count: int) -> None:
        print(f"Processed {count} orders")

    print_report(count_orders())


retail_hello()

This demonstrates Airflow’s TaskFlow API.

The decorated Python functions become Airflow tasks. Passing the result of count_orders() into print_report() creates their dependency.

Airflow handles the task relationship and passes the small returned value using its task communication mechanism.

For large datasets, pass references to files or tables between tasks rather than placing entire DataFrames in task metadata.

Start Airflow

Run:

airflow standalone

To run the DAG once from the terminal without starting the UI, use airflow dags test retail_hello. It executes both tasks in-process and prints their logs, which is handy for a first check.

The standalone command initializes the local environment and starts Airflow’s necessary components.

Open the UI at http://localhost:8080. Airflow prints local login information during initialization.

In another terminal, activate the same Airflow environment, set the same home directory, and trigger the DAG:

. .airflow-venv/bin/activate

export AIRFLOW_HOME="$(pwd)/airflow_home"

airflow dags trigger retail_hello

You should see two tasks in the DAG graph.

The second task runs after the first and prints the count to its task log.

The key lesson: Airflow orchestrates work. The work itself can be SQL, Python, Spark, dbt, or a remote service.

For scheduled production pipelines, task idempotency and backfill behavior are more important than the number of tasks in your diagram.

6. Prefect: a Python-first alternative for orchestration

What it is: Prefect provides workflow orchestration built around Python functions, flows, and tasks.

It can be convenient for smaller Python-oriented teams or workflows that benefit from dynamic control flow.

Use it when: You want to introduce task tracking, retries, and workflow structure while keeping execution close to normal Python.

Do not reach for it first when: You have no operational workflow requirements, or your organization already has a well-supported orchestrator that solves the problem.

Create a separate environment:

deactivate

python3.12 -m venv .prefect-venv
. .prefect-venv/bin/activate

python -m pip install "prefect==3.8.8"

Create prefect_hello.py:

from prefect import flow, task


@task(retries=2, retry_delay_seconds=1)
def read_orders() -> list[float]:
    return [25.0, 120.0, 15.0, 30.0]


@task
def calculate_revenue(amounts: list[float]) -> float:
    return sum(amounts)


@flow(name="retail-revenue")
def retail_revenue() -> None:
    amounts = read_orders()
    revenue = calculate_revenue(amounts)
    print(f"Total revenue: {revenue:.2f}")


if __name__ == "__main__":
    retail_revenue()

Run:

python prefect_hello.py

The result is a two-task workflow with a total revenue of 190.00.

Prefect adds orchestration behavior to decorated functions, including the configured retry behavior.

This local script does not need Prefect Cloud. You can also run the open-source Prefect server locally if you need a persistent UI, workflow history, or scheduled deployments.

Airflow versus Prefect: Both manage workflows. Airflow is a strong choice for established scheduled DAG environments; Prefect often feels natural when building Python-first workflows.

The choice should depend on operating requirements, deployment patterns, and team preferences—not which decorator looks shorter.

7. Apache Spark with PySpark: distributed data processing

What it is: Apache Spark is a distributed computing engine for large-scale data processing. PySpark provides its Python interface.

Spark can execute SQL, DataFrame transformations, joins, aggregations, and other processing workloads across machines.

It also supports running locally, which is how we will learn it.

Use it when: Processing demands distributed resources, your team already uses Spark infrastructure, or you need its broader large-scale processing ecosystem.

Do not reach for it first when: DuckDB, Polars, or pandas already complete the workload reliably on one machine.

Install PySpark

You need Java 17 or later, correctly configured on your machine. Spark 4.2 supports current Java runtimes including Java 17 and 21.

Create an environment:

deactivate

python3.12 -m venv .spark-venv
. .spark-venv/bin/activate

python -m pip install "pyspark==4.2.0"

java -version

No Docker container or Spark cluster is needed for this example.

Create a local Spark job

Save spark_hello.py:

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .appName("RetailHello")
    .master("local[2]")
    .getOrCreate()
)

orders = spark.createDataFrame(
    [
        (101, "books", 25.0),
        (102, "electronics", 120.0),
        (103, "books", 15.0),
        (104, "books", 30.0),
    ],
    schema="order_id INT, category STRING, amount DOUBLE",
)

orders.write.mode("overwrite").parquet("raw_orders_parquet")

loaded = spark.read.parquet("raw_orders_parquet")

sales = (
    loaded.groupBy("category")
    .agg(
        F.round(F.sum("amount"), 2).alias("revenue"),
        F.count("*").alias("order_count"),
    )
)

sales.show()

sales.write.mode("overwrite").parquet(
    "sales_by_category_parquet"
)

spark.stop()

Run:

python spark_hello.py

This program:

1. Starts a local Spark session using two worker threads.

2. Creates an orders DataFrame.

3. Writes the source data to Parquet.

4. Reads the Parquet data back.

5. Calculates sales by category.

6. Writes the aggregated result to Parquet.

Notice how familiar the transformation looks after learning pandas and Polars.

The difference is Spark’s execution model. Spark can construct distributed execution plans and coordinate tasks across workers.

In local mode, everything runs on one machine. That makes it useful for learning the API, but it does not reproduce every property of a production cluster.

Spark is powerful precisely because it solves distributed processing problems. That power introduces configuration, scheduling, resource management, and debugging complexity.

Do not assume those costs are justified by every data pipeline.

8. Apache Kafka: a durable stream of events

What it is: Kafka is a distributed event-streaming platform. Producers publish records to topics, and consumers read those records.

Kafka is frequently used to connect systems asynchronously, retain event history, and support streaming ingestion.

Use it when: Multiple applications must consume events, event volume is significant, or durable, replayable event distribution is part of the design.

Do not reach for it first when: A direct database write, simple batch import, or ordinary queue already satisfies your requirements.

Unlike pandas or DuckDB, Kafka requires a running broker.

We will use Docker for the broker. This is the only example in this guide that requires Docker.

Start a local broker

Run:

docker run -d \
  --name local-kafka \
  -p 9092:9092 \
  apache/kafka:4.2.2

The official Apache Kafka Docker image supports a single-broker local setup with default configuration.

After the broker finishes starting, create a topic:

docker exec local-kafka \
  /opt/kafka/bin/kafka-topics.sh \
  --create \
  --if-not-exists \
  --topic orders \
  --bootstrap-server localhost:9092

This creates an orders topic for our demonstration.

Produce and consume one event

Activate the original environment containing confluent-kafka, then save kafka_hello.py:

import json
import uuid

from confluent_kafka import Consumer, Producer

producer = Producer(
    {"bootstrap.servers": "localhost:9092"}
)

event = {
    "order_id": 101,
    "category": "books",
    "amount": 25.0,
}

producer.produce(
    topic="orders",
    key="101",
    value=json.dumps(event).encode("utf-8"),
)

remaining = producer.flush(timeout=10)

if remaining:
    raise RuntimeError("Message was not delivered")

consumer = Consumer(
    {
        "bootstrap.servers": "localhost:9092",
        "group.id": f"hello-{uuid.uuid4()}",
        "auto.offset.reset": "earliest",
        "enable.auto.commit": False,
    }
)

try:
    consumer.subscribe(["orders"])

    for _ in range(15):
        message = consumer.poll(timeout=1.0)

        if message is None:
            continue

        if message.error():
            raise RuntimeError(str(message.error()))

        decoded = json.loads(message.value().decode("utf-8"))
        print(decoded)
        break
    else:
        raise RuntimeError("No message received")
finally:
    consumer.close()

Run:

. .venv/bin/activate
python kafka_hello.py

You have now published an event to Kafka and consumed an event from the topic.

The unique consumer group ensures this demonstration starts with a fresh consumer position on every run. Because the topic retains earlier records, the consumed record might be from a previous execution.

That is an important Kafka behavior, not a bug.

Real Kafka consumers need additional handling for delivery guarantees, duplicate records, serialization formats, partitions, consumer offsets, and failure recovery.

Kafka is not the transformation engine. It moves and retains records; your consumers and stream-processing applications decide what to do with them.

9. Pandera: validate data before it becomes a problem

What it is: Pandera provides schema and validation capabilities for DataFrames, including pandas and other supported backends.

It allows you to define expected data types and validation rules directly in Python.

Use it when: Your pipelines depend on well-defined tabular inputs, and invalid rows or schemas should cause an explicit failure.

Do not reach for it first when: A few basic assertions are sufficient, or your data is already validated at the correct boundary by another reliable system.

Create quality_hello.py in the original environment:

import pandas as pd
import pandera.pandas as pa

orders = pd.DataFrame(
    {
        "order_id": [101, 102, 103],
        "amount": [25.0, 120.0, 15.0],
    }
)

schema = pa.DataFrameSchema(
    {
        "order_id": pa.Column(
            int,
            checks=pa.Check.ge(1),
            unique=True,
            nullable=False,
        ),
        "amount": pa.Column(
            float,
            checks=pa.Check.ge(0),
            nullable=False,
        ),
    },
    strict=True,
)

validated = schema.validate(orders)

print("Data quality checks passed")
print(validated)

Run:

python quality_hello.py

Try modifying one amount to a negative number and running it again.

The validation should fail instead of silently allowing the invalid record downstream.

The modern Pandera API uses the dedicated pandera.pandas import for pandas-specific schemas.

This is a small example, but the underlying principle is essential: bad data should fail as close to its source as practical, before it contaminates downstream reports and models.

In SQL-centric pipelines, dbt’s generic and custom tests can serve similar purposes. Pandera is particularly useful when transformations and their validation live in Python.

10. scikit-learn: train and evaluate a first model

What it is: scikit-learn provides machine learning algorithms and tools for preprocessing, training, evaluation, and model selection.

For many structured-data problems, it is the simplest sensible place to begin.

Use it when: You need classical machine learning for regression, classification, clustering, or similar tasks.

Do not reach for it first when: A simple business rule solves the problem, or your primary challenge requires a specialized deep learning or distributed training framework.

Our example: estimate delivery time

We will generate synthetic e-commerce delivery observations.

Each example has two features:

• Number of items in the order.

• Delivery distance in kilometers.

Our model predicts delivery time in minutes.

The synthetic target follows a simple relationship with random noise:

Estimated delivery time ≈ 8 + 2 × items + 1.5 × distance

This is only a teaching example. Real delivery predictions would need richer features, historical validation, and careful treatment of information available at prediction time.

Save train_delivery.py:

import joblib
import numpy as np

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(42)

n = 200

items = rng.integers(1, 10, size=n)
distance = rng.uniform(1.0, 25.0, size=n)

X = np.column_stack([items, distance])

noise = rng.normal(0, 1.5, size=n)
y = 8 + 2 * items + 1.5 * distance + noise

X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42,
)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)

mae = mean_absolute_error(y_test, predictions)
r2 = r2_score(y_test, predictions)

print(f"Mean absolute error: {mae:.2f} minutes")
print(f"R2 score: {r2:.3f}")

joblib.dump(model, "delivery_model.joblib")

print("Saved delivery_model.joblib")

Run:

python train_delivery.py

The script trains a linear regression model, evaluates it on held-out examples, and saves the fitted model.

Mean absolute error measures the average absolute difference between predicted and actual delivery times.

An MAE of 1.2 minutes means predictions differ from their targets by 1.2 minutes on average, for the evaluated dataset.

R2 measures how much variation the model explains relative to a constant baseline that predicts the target mean. Values closer to 1 generally indicate a better fit, though the metric can be negative.

For production work, you would also consider time-based validation where appropriate, prediction bias, error distributions, and changes in operating conditions.

Important: The saved joblib model uses Python object serialization. Only load model files from trusted sources, because loading untrusted serialized Python objects can execute malicious code.

11. MLflow: record experiments and model artifacts

What it is: MLflow provides experiment tracking, model packaging, registry capabilities, and broader ML lifecycle tooling.

For beginners, experiment tracking is the most immediately useful feature.

Use it when: You need to compare training runs, preserve their metrics and artifacts, or coordinate model development across iterations.

Do not reach for it first when: You have not established a meaningful training and evaluation process. Tracking experiments does not make a poorly designed experiment reliable.

We will use MLflow locally without a tracking server or cloud account.

Track a local experiment

Create mlflow_hello.py:

import mlflow
import mlflow.sklearn
import numpy as np

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

mlflow.set_tracking_uri("sqlite:///mlflow.db")

mlflow.set_experiment("delivery-estimator")

rng = np.random.default_rng(42)

X = rng.uniform(1, 20, size=(100, 1))
y = 5 + 2.5 * X[:, 0] + rng.normal(0, 1, size=100)

model = LinearRegression()
model.fit(X, y)

predictions = model.predict(X)
mae = mean_absolute_error(y, predictions)

with mlflow.start_run():
    mlflow.log_param("algorithm", "LinearRegression")
    mlflow.log_param("training_rows", len(X))
    mlflow.log_metric("training_mae", mae)

    model_info = mlflow.sklearn.log_model(
        model,
        name="delivery_model",
    )

    print(f"Logged model: {model_info.model_uri}")
    print(f"Training MAE: {mae:.3f}")

Run:

python mlflow_hello.py

MLflow stores experiment metadata and model artifacts under the local tracking directory.

The code uses the MLflow 3 model-logging interface, where the model is given a name and the returned model information provides its URI.

For this exercise, the file-based backend keeps everything local and easy to inspect. It is a compatibility backend in current MLflow and is in maintenance mode; a database-backed tracking store is recommended for serious long-term deployments.

Open the MLflow UI

From the same directory:

mlflow ui --backend-store-uri sqlite:///mlflow.db --port 5000

Why SQLite instead of a local ./mlruns folder? Since MLflow 3.17, the plain-file tracking backend is in maintenance mode, and pointing a new project at it raises an MlflowException asking you to use a database backend. A local SQLite file is the zero-setup database option; MLflow still stores the model artifacts under ./mlruns. If port 5000 is already taken on your machine (macOS AirPlay uses it), pick another port.

Open http://127.0.0.1:5000.

You can inspect the experiment, training run, logged parameters, metrics, and model artifacts.

Notice that we logged a training error, not an independent evaluation metric. For a real experiment, log held-out test metrics or cross-validation results separately.

That distinction prevents model performance from looking better than it really is.

What about the model registry?

Experiment tracking and registration serve different purposes.

In a more complete workflow, models that pass evaluation can be registered and assigned versioned identities. Deployment systems then select the intended registered version or alias.

You do not need to create a registry just to save one local model.

Start with reproducible training and meaningful evaluation. Add registry governance when multiple model versions and environments make it valuable.

12. FastAPI: serve a trained model through HTTP

What it is: FastAPI is a Python web framework for building HTTP APIs, including prediction services.

It uses type annotations and Pydantic models to validate incoming requests.

Use it when: An application needs predictions on demand, and exposing an HTTP endpoint is an appropriate integration pattern.

Do not reach for it first when: Predictions only need to run in a daily batch job, or you do not need a continuously available service.

We will serve the delivery-time model created by the scikit-learn example.

Make sure delivery_model.joblib exists by running train_delivery.py first.

Save serve_delivery.py:

import joblib
import numpy as np

from fastapi import FastAPI
from pydantic import BaseModel, Field

app = FastAPI(title="Delivery Time Estimator")

model = joblib.load("delivery_model.joblib")


class DeliveryRequest(BaseModel):
    items: int = Field(ge=1, le=100)
    distance_km: float = Field(ge=0, le=500)


class DeliveryResponse(BaseModel):
    estimated_minutes: float


@app.get("/")
def home():
    return {"status": "ok"}


@app.post("/predict", response_model=DeliveryResponse)
def predict(request: DeliveryRequest):
    features = np.array(
        [[request.items, request.distance_km]],
        dtype=float,
    )

    prediction = float(model.predict(features)[0])

    return DeliveryResponse(
        estimated_minutes=round(prediction, 2)
    )

Start the server:

uvicorn serve_delivery:app --host 127.0.0.1 --port 8000

In a second terminal, send a prediction request:

curl -X POST "http://127.0.0.1:8000/predict" \
  -H "Content-Type: application/json" \
  -d '{"items": 3, "distance_km": 8.5}'

You should receive JSON containing an estimated delivery time.

You can also visit http://127.0.0.1:8000/docs to use FastAPI’s interactive API documentation.

What did we just build?

The workflow now contains three distinct steps:

1. scikit-learn trained the prediction model.

2. joblib saved the trained artifact.

3. FastAPI loaded the artifact and made predictions available over HTTP.

MLflow could separately track the model’s development runs and support model version management.

This separation of responsibilities is the foundation of practical ML serving.

The endpoint is deliberately minimal. A production service would need appropriate authentication, access controls, observability, release management, concurrency testing, and model version governance.

Also notice that request validation allows a broader range than the synthetic training data. Real systems should define which inputs are operationally valid and monitor predictions outside the training distribution.

Part 4: The tools fit together — but not every project needs all of them

The most common beginner mistake is thinking that a modern data stack consists of installing every popular framework.

It is more useful to group tools by responsibility.

This explains why some tools appear together frequently.

For example, Airflow and Spark are not alternatives. Airflow can schedule Spark jobs.

Likewise, dbt and DuckDB are complementary. dbt manages SQL transformations, while DuckDB executes them.

And MLflow does not replace scikit-learn. The former tracks and manages experiments and models; the latter can train them.

A practical sequence for our imaginary shop

The shop might adopt tools gradually.

Stage 1: Basic analytics

Start with CSV files, pandas, and DuckDB.

Write transformations, produce reports, and verify that the results answer useful business questions.

Stage 2: Repeatable reporting

Add dbt when SQL models accumulate. Add validation checks and a scheduler when reports become operational responsibilities.

A simple operating-system scheduler might still be enough.

Stage 3: More reliable workflows

Introduce Prefect or Airflow when jobs have dependencies, retry requirements, backfills, and monitoring needs.

Stage 4: Larger data workloads

Evaluate Polars, DuckDB, and efficient Parquet processing first. Move to Spark when measurements show a distributed engine is necessary.

Stage 5: Event-driven requirements

Adopt Kafka when durable event distribution and independent event consumers create genuine value.

Stage 6: Machine learning

Train and evaluate models with scikit-learn. Add MLflow to track experiments. Use FastAPI when the application needs online inference.

Each stage should solve a real bottleneck.

You do not need to implement all six stages simultaneously.

Part 5: Decision table — which tool should you reach for first?

The following table is designed as a practical reference. Start with the simpler choice, and graduate when a requirement or measured limitation justifies the change.

There is no universal row-count threshold at which you must abandon pandas or DuckDB for Spark.

The correct threshold depends on memory, storage bandwidth, query patterns, runtime requirements, workload concurrency, and how much infrastructure you are willing to operate.

Benchmark representative workloads before making the decision.

Part 6: Cherry on the cake — when DuckDB was the better choice than Spark

A particularly useful real-world story comes from Digital Turbine.

In its engineering article, Small-Scale Data Pipeline: DuckDB vs. chDB vs. Spark, published March 25, 2025, engineer Daniel Krupitsky described an aggregation workload associated with the company’s advertising platform.

The team was familiar with Spark, but wanted a simpler way to run a frequent aggregation (every few minutes) over approximately 40 GiB of input data.

The benchmark used 52 Parquet files on a machine with 16 CPU cores and 64 GB of RAM.

The published results were striking:

Source: Digital Turbine engineering benchmark, March 2025.

DuckDB completed the tested task in 3.99 seconds, compared with Spark’s 16.39 seconds under the reported configuration.

The team chose DuckDB for that particular workload, which could be scheduled with a Kubernetes CronJob. Importantly, it continued using Spark elsewhere, where distributed processing remained valuable.

The story is not evidence that DuckDB is always faster than Spark. It is evidence that the best tool depends on the workload.

A cluster-oriented framework introduces overhead, even when you run it locally. For a compact analytical job, that overhead may outweigh its advantages.

The broader lesson is about engineering discipline:

Do not choose infrastructure based solely on the largest scale your organization might someday reach. Choose for the workload you can measure, while keeping a sensible growth path.

That approach often produces systems that are easier to understand, less expensive to operate, and faster to change.

Part 7: Five engineering habits that matter more than tool selection

The hello worlds above demonstrate basic APIs, but production reliability comes from decisions surrounding the code.

1. Make reruns safe

A pipeline should anticipate retries and duplicate execution.

For example, a daily revenue job should replace or upsert the correct reporting period rather than blindly append another set of totals.

Track stable business keys and make output ownership explicit.

2. Keep transformations testable

Break transformations into logical stages with clear input and output contracts.

Test important business rules independently from scheduling infrastructure.

A failed calculation should be reproducible without requiring an entire orchestration platform.

3. Preserve raw information where justified

Cleaning data sometimes destroys information you later need for debugging or recomputation.

Retaining raw inputs, subject to privacy and retention requirements, can make backfills and investigations much easier.

The storage decision should still reflect cost, access controls, and the business value of historical records.

4. Measure cost and operational complexity

Compute time is not the only cost.

Also consider:

• Infrastructure provisioning and maintenance.

• Engineering time spent debugging.

• Cluster startup overhead.

• Data movement and storage.

• Reliability and recovery requirements.

• Onboarding new team members.

A workload that runs slightly faster on a complex platform may still be more expensive overall.

5. Separate experiments from operational systems

A notebook is a good place to explore ideas. A scheduled pipeline or prediction service needs a reproducible and observable execution path.

Keep configuration explicit, dependencies isolated, and model artifacts versioned.

Use suitable secrets management and authentication when integrations become external or production-facing.

Part 8: A hands-on learning path

To turn this guide into practical experience, build one small end-to-end retail analytics project and extend it incrementally.

Exercise A: Build a sales report

Use pandas or Polars to calculate revenue by category, then recreate the calculation in DuckDB.

Compare the code and results. The goal is to understand two styles of expressing the same business transformation.

Exercise B: Turn SQL into a tested project

Use the dbt example to add another mart, such as revenue by category and date.

Add a test that ensures the reporting date is never null, and deliberately insert an invalid record to see how validation behaves.

Exercise C: Automate the workflow

Create a Prefect flow or an Airflow DAG that prepares source data, runs validation, and executes the reporting transformation.

Rerun the workflow to check whether the output is idempotent.

Exercise D: Train and serve the delivery model

Use the scikit-learn example, record a run with MLflow, and serve the model through FastAPI.

Make several prediction requests and inspect how item count and distance influence the output.

Then consider which additional features would be useful in a real system and whether they could be obtained at prediction time.

Exercise E: Explain your architecture

Draw your own diagram and answer five questions:

• Where does the data originate?

• Which component stores it?

• Which component performs transformations?

• Which component manages execution?

• How are correctness and failures detected?

If you can answer those questions clearly, you have a much stronger grasp of data engineering architecture than someone who has merely installed a dozen tools.

Final takeaway: Start with the problem, not the platform

The data and machine learning ecosystem becomes much less intimidating once the responsibilities are separated.

pandas and Polars manipulate DataFrames. DuckDB runs local analytical SQL. dbt organizes SQL transformations. Airflow and Prefect orchestrate workflows. Spark distributes computation. Kafka distributes events. Pandera validates DataFrames. scikit-learn trains models. MLflow manages experiment information and model artifacts. FastAPI exposes predictions.

These tools can complement one another, but they do not all belong in every project.

For a beginner, the most valuable progression is to build something useful with Python and SQL, add tests, make it repeatable, and introduce infrastructure only when a real requirement demands it.

Your next step is concrete: choose the retail-orders example from this guide, build a local DuckDB report, convert it into a tested dbt project, and then add one orchestrated workflow. Once that works, extend it with a tracked ML experiment and a FastAPI prediction endpoint.

You will finish with more than a collection of hello worlds. You will have a working foundation for understanding, designing, and growing real data and ML systems.