← All courses

✨ Apache Spark

What changes when one machine is genuinely not enough — DataFrames and the Catalyst plan, partitions and shuffle, joins that skew, and an honest answer to when a cluster is the wrong tool.

No lessons published yet — this is the roadmap, so you can see exactly what the course will cover before it lands.

Before you start: Learn SQL — Series 1 and Series 2. Spark is a SQL engine with a DataFrame API on top. Every optimisation lesson here is one about joins, grouping and cardinality, so you need those before the distribution part makes any sense.

Short on time? The 13 episodes marked core are the shortest path to being productive at work. Start there and pick up the rest whenever a job asks for them.

Series 1 · Spark Fundamentals

whether you need it, and the model if you do

  1. 01

    Do You Need Spark — and the Three Cheaper Answers to Try First

    coreintermediatecoming soon
  2. 02

    Driver, Executors, Partitions — and Why an Action Re-Runs Everything

    coreintermediatecoming soon
  3. 03

    DataFrames and Spark SQL — Two Syntaxes, One Engine

    coreintermediatecoming soon
  4. 04

    Reading and Writing — Partitioned Layouts, and the Small-Files Problem You Just Made

    coreintermediatecoming soon
  5. 05

    UDFs — Why a Python UDF Is a Cliff, and What pandas UDFs Fix

    intermediatecoming soon
  6. 06

    Project — A Job Over Partitioned Parquet That Reads Only What It Needs

    coreintermediate#projectcoming soon

Series 2 · Making It Fast

the shuffle, the skew, and reading the UI

  1. 01

    The Shuffle — What Causes It, and Why It Is Almost Always the Slow Part

    coreadvanced#dean-mapreducecoming soon
  2. 02

    The Spark UI and explain() — Finding Which Stage Is Actually Slow

    coreadvancedcoming soon
  3. 03

    Join Strategies — Broadcast the Small Side, and When Spark Will Not

    coreadvancedcoming soon
  4. 04

    Data Skew — One Task Running for an Hour While the Other 199 Finished

    coreadvancedcoming soon
  5. 05

    Caching and Adaptive Query Execution — What You No Longer Have to Tune

    advancedcoming soon
  6. 06

    Project — Take a Slow Job and Halve It, With the UI as Evidence

    coreadvanced#projectcoming soon

Series 3 · In Production

tables, streaming, and an honest verdict

  1. 01

    Delta and Iceberg From Spark — MERGE INTO, Time Travel and Compaction

    coreadvancedcoming soon
  2. 02

    Structured Streaming — The Same API Over an Unbounded Table, in Micro-Batches

    advancedcoming soon
  3. 03

    spark-submit, Configuration and Tests — Handing a Job to Someone Else

    advancedcoming soon
  4. 04

    When Spark Wins — Finding the Crossover Point Instead of Assuming It

    coreadvancedcoming soon
  5. 05

    Project — A Job That Reads a Lakehouse Table, Reconciles, and Writes One Back

    coreadvanced#projectcoming soon

Not here, on purpose

  • SQL itself — covered in Learn SQL
  • Running and sizing a cluster — YARN, Kubernetes, autoscaling, cost — covered in Data Ops for the operational discipline. Here you only learn what a JOB AUTHOR must know about executors and partitions
  • The RDD API — not covered beyond recognising it. The DataFrame API is what you write today
  • MLlib and machine learning — not covered; this track moves and shapes data
  • Streaming concepts — event time, watermarks, windows — covered in ETL & Data Pipelines, Series 3 · 42 and in Apache Flink. In this course, Series 3 · 05 is Spark's implementation, not the theory