🔄 ETL & Data Pipelines

ETL vs. ELT, staging, idempotent and incremental loads with CDC, and orchestrating the whole flow.

No lessons published yet — this is the roadmap, so you can see exactly what the course will cover before it lands.

Before you start: Learn SQL — Series 1. Every transform here is a SELECT and every check is a COUNT. You do not need clever SQL, but you need to know what the right answer looks like before a pipeline tells you.

Short on time? The 28 episodes marked core are the shortest path to being productive at work. Start there and pick up the rest whenever a job asks for them.

Series 1 · ETL Fundamentals

build a pipeline that loads data and is safe to run twice

01
What a Pipeline Is — Extract, Transform, Load, and Where Each One Happens
corebeginnercoming soon
02
Your First Pipeline — Load a CSV into a Table
corebeginnercoming soon
03
Your First Export — Write a Table Out as a File Someone Else Can Load
corebeginnercoming soon
04
Sources & Sinks — Files, APIs and Databases as Connections
corebeginnercoming soon
05
Staging — Land Raw Before You Transform Anything
corebeginnercoming soon
06
Types & Casting Across Sources — When 'Y', 'yes' and 1 All Mean True
corebeginnercoming soon
07
Transform with SQL — Reshaping the Data on Its Way In
corebeginnercoming soon
08
Idempotency — Re-Runnable Loads with TRUNCATE+INSERT and MERGE
corebeginnercoming soon
09
Incremental Loads & Watermarks — Only What Changed
corebeginnercoming soon
10
Row Counts & Logging — Proving the Load Did What You Think
corebeginnercoming soon
11
From Script to Scheduled Job — Hand a Pipeline to the Scheduler
corebeginnercoming soon
12
Project — A Load That Is Safe to Run Twice
corebeginner#projectcoming soon

Series 2 · Pipelines You Can Maintain

handle bad rows, late data and failures without hand-fixing

01
Scripting a Pipeline — When a Script Beats a Chain of SQL Files
intermediatecoming soon
02
Parameterized & Reusable Jobs — One Script, Many Tables
intermediatecoming soon
03
Schema Drift — When the Source Changes Shape and Every Row Still Loads
coreintermediatecoming soon
04
Data Quality Gates — Nulls, Ranges and Referential Gaps, Before They Land
coreintermediatecoming soon
05
Testing a Pipeline — Catching a Bad Change Before Tonight's Run Does
coreintermediatecoming soon
06
Deduplication & Conformance — One Customer, One Row
coreintermediatecoming soon
07
Late & Out-of-Order Data — Rows That Arrive After the Run That Wanted Them
coreintermediate#kimballcoming soon
08
Errors, Retries & Dead Letters — Fail Without Losing Rows
coreintermediatecoming soon
09
Personal Data — Masking, Synthetic Rows and a Safe Copy of Production
intermediatecoming soon
10
Surrogate Keys & Dimension Lookups — Business Keys to Warehouse IDs
coreintermediate#kimballcoming soon
11
Load Order — Dimensions First, Then Facts, and What Breaks Otherwise
coreintermediate#kimballcoming soon
12
Cross-Database Pipelines — Northwind PostgreSQL → ClickHouse
intermediatecoming soon
13
Orchestration & DAGs — Expressing Dependencies Instead of Chaining Cron Jobs
coreintermediatecoming soon
14
Project — One Job That Loads Five Tables in the Right Order
coreintermediate#projectcoming soon

Series 3 · Loading at Production Volume

load a star schema, track changes, and re-run a whole month

01
ETL vs ELT — Where the Transform Should Run, Now That You Have Done Both
coreadvancedcoming soon
02
Building the Star Schema — Loading One From SQLite Into DuckDB and ClickHouse
coreadvanced#kimballcoming soon
03
SCD Type 2 Loads — Keep History Without Duplicating It
coreadvanced#kimballcoming soon
04
Change Data Capture — Reading the Database's Own Log, Deletes Included
advancedcoming soon
05
Backfills & Reprocessing — Rewrite History Without Breaking Downstream
coreadvancedcoming soon
06
Idempotent MERGE at Scale — and the Vendor Differences That Break It
advancedcoming soon
07
Data Contracts, Lineage & Observability — Know What Broke, and Who Depends On It
advanced#kimballcoming soon
08
Bulk Loading & Parallelism — COPY, Partitions and Throughput
advancedcoming soon
09
Streaming vs Batch — Is Your Data Bounded, and Do You Need a Stream?
coreadvancedcoming soon
10
Project — Orchestrate the Full Northwind Pipeline End to End
coreadvanced#projectcoming soon

Series 4 · Streaming Pipelines

the same correctness, when the data never ends

01
Event Time, Windows and Watermarks — Deciding When an Answer Is Final
advancedcoming soon
02
Staging in a Stream — The Log Is Your Raw Layer, and It Expires
advancedcoming soon
03
Bad Records Without a Human — Quarantine Instead of Stopping the Pipeline
advancedcoming soon
04
Deduplication With State — One Customer, One Row, Within Seven Days
advancedcoming soon
05
The Fact That Arrived Before Its Dimension — Load Order You Do Not Control
advancedcoming soon
06
Dimension Lookups in Flight — Joining to a Table That Is Changing Under You
advancedcoming soon
07
Idempotent Sinks — Why Exactly-Once Stops at the Edge of the Stream
advancedcoming soon
08
Reconciling a Stream — Counting Something That Has No End
advancedcoming soon
09
Replay and Reprocessing — Rewriting History Without Double-Counting It
advancedcoming soon
10
Project — A Streaming Pipeline That Is Safe to Replay
advanced#projectcoming soon

Where this leads

Analytics Engineering with dbt — starts from Series 1.
Apache Airflow — starts from Series 1, and Series 2 · 45.
Apache Beam — starts from Series 3 · 42.
Apache Flink — starts from Series 3 · 42.
Apache Kafka — starts from Series 1, and Series 3 · 42.
Build Your Own: Algotrading System — starts from Series 1.

Not here, on purpose

The SQL inside the transforms — covered in Learn SQL
Designing the target schema — covered in Data Modeling
Doing the T as version-controlled, tested models — covered in Analytics Engineering with dbt — this track owns Extract and Load
cron internals, monitoring, alerting and on-call — covered in Data Ops — here a pipeline just gets handed to the scheduler
Where the credentials live — environment variables, secret stores, and file permissions — covered in Data Ops, Series 1, along with telling a credentials failure apart from a firewall one. In this course a connection is something you are handed, already configured
Learning the language you write the pipeline in — covered in Python for Data and Java & Groovy for Data — either one; the pipeline shape here is the same in both
The warehouse the data lands in — engines, partitions, compression — covered in Data Warehousing — here it is a destination you load into
Deciding what the star schema should be — grain, SCD type, star vs snowflake — covered in Data Modeling, Series 3. In this course, Series 3 · 05 loads a star that somebody has already designed
Choosing and running the scheduler itself — cron vs systemd vs an orchestrator — covered in Data Ops, Series 3, which is choosing and running the scheduler. In this course, Series 2 · 45 is expressing your pipeline as a DAG, whatever ends up running it

🔄 ETL & Data Pipelines

Series 1 · ETL Fundamentals

What a Pipeline Is — Extract, Transform, Load, and Where Each One Happens

Your First Pipeline — Load a CSV into a Table

Your First Export — Write a Table Out as a File Someone Else Can Load

Sources & Sinks — Files, APIs and Databases as Connections

Staging — Land Raw Before You Transform Anything

Types & Casting Across Sources — When 'Y', 'yes' and 1 All Mean True

Transform with SQL — Reshaping the Data on Its Way In

Idempotency — Re-Runnable Loads with TRUNCATE+INSERT and MERGE

Incremental Loads & Watermarks — Only What Changed

Row Counts & Logging — Proving the Load Did What You Think

From Script to Scheduled Job — Hand a Pipeline to the Scheduler

Project — A Load That Is Safe to Run Twice

Series 2 · Pipelines You Can Maintain

Scripting a Pipeline — When a Script Beats a Chain of SQL Files

Parameterized & Reusable Jobs — One Script, Many Tables

Schema Drift — When the Source Changes Shape and Every Row Still Loads

Data Quality Gates — Nulls, Ranges and Referential Gaps, Before They Land

Testing a Pipeline — Catching a Bad Change Before Tonight's Run Does

Deduplication & Conformance — One Customer, One Row

Late & Out-of-Order Data — Rows That Arrive After the Run That Wanted Them

Errors, Retries & Dead Letters — Fail Without Losing Rows

Personal Data — Masking, Synthetic Rows and a Safe Copy of Production

Surrogate Keys & Dimension Lookups — Business Keys to Warehouse IDs

Load Order — Dimensions First, Then Facts, and What Breaks Otherwise

Cross-Database Pipelines — Northwind PostgreSQL → ClickHouse

Orchestration & DAGs — Expressing Dependencies Instead of Chaining Cron Jobs

Project — One Job That Loads Five Tables in the Right Order

Series 3 · Loading at Production Volume

ETL vs ELT — Where the Transform Should Run, Now That You Have Done Both

Building the Star Schema — Loading One From SQLite Into DuckDB and ClickHouse

SCD Type 2 Loads — Keep History Without Duplicating It

Change Data Capture — Reading the Database's Own Log, Deletes Included

Backfills & Reprocessing — Rewrite History Without Breaking Downstream

Idempotent MERGE at Scale — and the Vendor Differences That Break It

Data Contracts, Lineage & Observability — Know What Broke, and Who Depends On It

Bulk Loading & Parallelism — COPY, Partitions and Throughput

Streaming vs Batch — Is Your Data Bounded, and Do You Need a Stream?

Project — Orchestrate the Full Northwind Pipeline End to End

Series 4 · Streaming Pipelines

Event Time, Windows and Watermarks — Deciding When an Answer Is Final

Staging in a Stream — The Log Is Your Raw Layer, and It Expires

Bad Records Without a Human — Quarantine Instead of Stopping the Pipeline

Deduplication With State — One Customer, One Row, Within Seven Days

The Fact That Arrived Before Its Dimension — Load Order You Do Not Control

Dimension Lookups in Flight — Joining to a Table That Is Changing Under You

Idempotent Sinks — Why Exactly-Once Stops at the Edge of the Stream

Reconciling a Stream — Counting Something That Has No End

Replay and Reprocessing — Rewriting History Without Double-Counting It

Project — A Streaming Pipeline That Is Safe to Replay

Where this leads

Not here, on purpose