-
Orchestration in 2026: Airflow assets and dynamic task mapping vs AWS Step Functions and Lambda, and what to pick for data vs ML
Modern workflow orchestration is no longer just about scheduling a Python script at midnight. Today’s data platforms must coordinate warehouse transformations, react to incoming files, handle failures
-
The data and ML tool map: hello worlds for dbt, Airflow, Spark, MLflow and friends, and when to use each
If you are new to data engineering or machine learning, the modern tool landscape can feel unnecessarily complicated.
-
Data versioning and lineage: reproducing last month’s training set exactly
Imagine this production incident.
-
Medallion architecture: bronze, silver, gold, and when layers are just bureaucracy
A data pipeline can have bronze, silver, and gold tables, beautifully organized schemas, dozens of scheduled jobs, and still produce numbers nobody trusts.
-
Data quality as code: contracts, expectations, and stopping bad data at the door
A data pipeline can run successfully, pass every scheduled job, update every dashboard, and still produce completely incorrect business decisions.
-
Streaming data: Kafka, exactly-once, and why it’s harder than it sounds
Imagine an e-commerce platform processing thousands of orders every second. A customer places an order, an event enters a Kafka topic, and a consumer processes that event to create a purchase record.
-
The lakehouse: Iceberg, Delta, and tables that time-travel
A plain object-storage data lake is wonderfully simple: put Parquet files in S3, Azure Data Lake Storage, Google Cloud Storage, HDFS, or another filesystem-like service, then point query engines at th
-
Orchestration beyond cron: DAGs, retries, and backfills with Airflow or Dagster
Cron is excellent at one job: starting a command at a particular time.
-
ELT and dbt: transforming data where it lives, with tests and lineage
ELT changes the center of gravity of a data pipeline.
-
Polars vs pandas: the DataFrame rewrite that changed the default
For years, the default choice for tabular Python was automatic: import pandas and start manipulating rows. That default has changed.