← All courses

🌬️ Apache Airflow

Orchestration as code — DAGs, scheduling and backfills, sensors and dependencies, retries and idempotent tasks, and what an orchestrator buys over a cron entry.

No lessons published yet — this is the roadmap, so you can see exactly what the course will cover before it lands.

Before you start: ETL & Data Pipelines — Series 1, and Series 2 · 45. You need a pipeline worth orchestrating, and you need idempotency — because every backfill and every retry in this course is safe only if the load underneath is re-runnable. ETL & Data Pipelines Series 1 · 30 is that lesson and this track assumes it.

Short on time? The 15 episodes marked core are the shortest path to being productive at work. Start there and pick up the rest whenever a job asks for them.

Series 1 · DAGs and Scheduling

the model, and the date that is not today

  1. 01

    DAGs and Tasks — The File That Is Parsed Every Thirty Seconds

    coreintermediatecoming soon
  2. 02

    The Logical Date — Why the Run for Monday Happens on Tuesday, and now() Is a Bug

    coreintermediatecoming soon
  3. 03

    Schedules, catchup and Backfills — the Hundred Runs You Did Not Mean to Start

    coreintermediatecoming soon
  4. 04

    Dependencies and Trigger Rules — Branching, Cleanup, and Why Downstream Skipped

    coreintermediatecoming soon
  5. 05

    XComs Carry Pointers, Not Data — and Where Credentials Belong

    coreintermediatecoming soon
  6. 06

    Project — A DAG That Is Safe to Re-Run for Any Day Last Month

    coreintermediate#projectcoming soon

Series 2 · Authoring Well

orchestrate, do not compute

  1. 01

    Choosing an Operator — Make Something Else Do the Work

    coreadvancedcoming soon
  2. 02

    Sensors — The Hundred Pokes That Filled Every Worker Slot

    coreadvancedcoming soon
  3. 03

    Retries, Timeouts and Alerts — and What a Retry Does to a Non-Idempotent Task

    coreadvancedcoming soon
  4. 04

    Dynamic Task Mapping — One Task Per File, Decided at Runtime

    advancedcoming soon
  5. 05

    Datasets — Running When the Data Is Ready Instead of Guessing a Time

    coreadvancedcoming soon
  6. 06

    Testing DAGs — Keep the Logic Outside Airflow So It Can Be Tested at All

    advancedcoming soon
  7. 07

    Project — Two DAGs Where One Runs Because the Other Produced Something

    coreadvanced#projectcoming soon

Series 3 · Running It

what an author must know about the machinery

  1. 01

    Executors and Pools — Where Your Task Runs, and Stopping One DAG Starving the Rest

    coreadvancedcoming soon
  2. 02

    Debugging a Run — Task States, Zombies, and Clearing to Re-Run Part of a DAG

    coreadvancedcoming soon
  3. 03

    Is Airflow Worth It — the Cron Entry That Would Have Done, and the Alternatives

    coreadvancedcoming soon
  4. 04

    Project — Take a Fragile Chain of Cron Jobs and Express It as One DAG

    coreadvanced#projectcoming soon

Not here, on purpose

  • Whether to use an orchestrator at all, and which one — covered in Data Ops, Series 3 · 00, which chooses between cron, systemd timers and an orchestrator
  • Idempotency and re-runnable loads — covered in ETL & Data Pipelines, Series 1 · 30. Assumed here — every backfill depends on it
  • Deploying and operating Airflow — executors at scale, Kubernetes, upgrades, HA — covered in Data Ops for the operational discipline. In this course, Series 3 · 00 covers only what an author must know about the executor
  • The work the tasks actually do — covered in ETL & Data Pipelines, Analytics Engineering with dbt, and the engine tracks. A DAG orchestrates; it does not transform