← All courses
💽 Storage for Data
The layer underneath everything else — block, file and object as three interfaces; IOPS against throughput against latency; RAID, replication and erasure coding, and why none of them is a backup; ZFS and the checksums that can tell you the data is wrong; the S3 protocol and hosting it yourself when the data cannot leave the building; and Ceph.
No lessons published yet — this is the roadmap, so you can see exactly what the course will cover before it lands.
Before you start: Data Ops — Series 1. You need the terminal, and you need to have already met the two episodes this track goes underneath — the machine with a disk that fills, and the bucket you rented from somebody else. This course is what both of those are standing on.
Short on time? The 40 episodes marked core are the shortest path to being productive at work. Start there and pick up the rest whenever a job asks for them.
Series 1 · What Storage Is, and the Linux Stack On Top
three interfaces, four numbers, and the layers between a disk and a directory
- 01
Block, File and Object — Three Interfaces, and the Question That Sorts Them
coreintermediatecoming soon - 02
IOPS, Throughput, Latency and Queue Depth — Four Numbers, Not One
coreintermediatecoming soon - 03
fio — Measuring the Four Numbers Instead of Reading Them
coreintermediatecoming soon - 04
NVMe, SSD, Spinning Disk and Network Storage — Ratios That Outlive the Prices
coreintermediatecoming soon - 05
lsblk, GPT and by-id — Looking at a Disk Before Anything Owns It
coreintermediatecoming soon - 06
RAID, Replication and Erasure Coding — What Each One Survives, and What None Does
coreintermediatecoming soon - 07
mdadm — the RAID Levels as Commands, and the Question It Cannot Answer
coreintermediatecoming soon - 08
LVM — PVs, VGs and LVs, and Growing a Filesystem While It Is Mounted
coreintermediatecoming soon - 09
ext4, XFS and fstab — Formatting, Mounting, and the One That Cannot Shrink
coreintermediatecoming soon - 10
When a Disk Dies — Rebuild Time, the Storm, and the Second Failure
coreintermediatecoming soon - 11
The 80% Cliff, Growth Rate and Tiering — and the Restore Time You Agreed To
intermediatecoming soon - 12
NFS — the Shared Filesystem This Academy Keeps Telling You to Use
intermediatecoming soon - 13
Encryption at Rest — What It Protects Against, and Who Holds the Key
intermediatecoming soon - 14
Project — Size and Cost the Storage for a Real Workload, and Defend It
coreintermediate#projectcoming soon
Series 2 · Filesystems, and ZFS
what a filesystem owes a database, and the one that can prove it
- 01
Journalling, Copy-on-Write and fsync — What a Database Needs From a Filesystem
coreadvancedcoming soon - 02
ZFS — What It Is, Why It Exists, and When It Earns Its Place
coreadvancedcoming soon - 03
Pools, vdevs and Datasets — Why Redundancy Lives in the vdev
coreadvancedcoming soon - 04
Mirrors, RAIDZ or Striped Mirrors — Exactly the Redundancy You Need
coreadvancedcoming soon - 05
Checksums and Scrub — the Filesystem That Can Tell You the Data Is Wrong
coreadvancedcoming soon - 06
zpool replace — the Command for the Thing That Will Actually Happen
coreadvancedcoming soon - 07
Snapshots, Clones and send/receive — Copies That Cost Nothing Until They Change
coreadvancedcoming soon - 08
recordsize, Compression and the ARC — the Settings That Matter, and One That Lies
advancedcoming soon - 09
btrfs — Copy-on-Write in the Kernel, and the Flag That Turns Off Checksums
advancedcoming soon - 10
Do You Need ZFS — and the Simpler Things That Would Have Worked
coreadvancedcoming soon - 11
Project — Build a Pool, Corrupt a Disk on Purpose, and Watch It Repair
coreadvanced#projectcoming soon
Series 3 · Object Storage — the Protocol, and Running It Yourself
what the API promises, and the same thing on your own disks
- 01
A Flat Key Space — Why Listing Is a Scan and There Are No Directories
coreintermediatecoming soon - 02
Multipart, Range GET and Presigned URLs — the Operations That Make It Usable
coreintermediatecoming soon - 03
Consistency, Versioning and Object Lock — What the Store Actually Promises
coreintermediatecoming soon - 04
Lifecycle Rules and Storage Classes — Tiering as Configuration, and Its Trap
intermediatecoming soon - 05
When the Data Cannot Leave — Four Reasons, and the Cost Model Inverted
coreintermediatecoming soon - 06
MinIO — the Same API, on Four Directories, in Ten Minutes
coreintermediatecoming soon - 07
Users, Policies and TLS — Securing a Bucket You Host Yourself
coreintermediatecoming soon - 08
Erasure Sets and Drive Layout — the Parity You Chose, as a Deployment
coreintermediatecoming soon - 09
Monitoring and Replication — an Alert, and a Copy That Survives a Delete
coreintermediatecoming soon - 10
What You Now Own — Capacity, Upgrades, and the Backup Nobody Else Is Taking
intermediatecoming soon - 11
Project — Host a Bucket, Point a Pipeline At It, and Read One Column
coreintermediate#projectcoming soon
Series 4 · Ceph
one cluster, three interfaces, and the algorithm underneath
- 01
One Cluster, Three Interfaces — RGW, RBD and CephFS on One Pool of Disks
coreadvancedcoming soon - 02
CRUSH — Computing Where the Data Lives Instead of Looking It Up
coreadvancedcoming soon - 03
Sizing and Two Networks — Why Ceph Is Network-Bound Before It Is Disk-Bound
coreadvancedcoming soon - 04
cephadm — Bootstrapping a Cluster, and What Healthy Looks Like
coreadvancedcoming soon - 05
cephx, Users and Keyrings — the Credential Every Client Needs First
coreadvancedcoming soon - 06
Pools and Failure Domains — Saying 'Survive a Rack' in Configuration
coreadvancedcoming soon - 07
RGW — the S3 Endpoint, and the Same Client Pointed at It
coreadvancedcoming soon - 08
RBD — Block Devices, and What a Database on Them Feels
advancedcoming soon - 09
CephFS — a Shared Filesystem, and When You Are Paying for POSIX You Do Not Use
advancedcoming soon - 10
ceph status, Recovery and Backfill — and Whether Anyone Will Own This
coreadvancedcoming soon - 11
Upgrades — the Order That Is Not Negotiable, and the Flag You Must Clear
advancedcoming soon - 12
HEALTH_WARN — the Warnings You Will Meet, and Which Ones Are Normal
coreadvancedcoming soon - 13
Rook — What Answers a PersistentVolumeClaim, and What Should
advancedcoming soon - 14
Do You Need Ceph — and What Would Have Been Enough
coreadvancedcoming soon - 15
Project — Choose a Storage Layer for Three Workloads, With the Numbers
coreadvanced#projectcoming soon
Where this leads
- Virtualization for Data — starts from Series 1.
Not here, on purpose
- Renting object storage, its bill, and using it in a job — covered in Data Ops, Series 1 · 47, which is the bucket a data person rents and pays for. In this course, Series 3 is the protocol itself and running one yourself
- Backups as a scheduled job, rotation and proving a restore — covered in Data Ops, Series 1 · 42. In this course, Series 1 · 20 explains what replication does and does not survive, which is the mechanism under that episode's rule
- Columnar layout, Parquet internals and compression for analytics — covered in Data Warehousing, Series 2. This track ends at the bytes; how you arrange rows inside a file to be skipped is that course's subject
- Each engine's own storage engine — covered in the engine tracks — ClickHouse for MergeTree, MySQL & MariaDB for InnoDB, Apache Cassandra for SSTables, PostgreSQL for the heap and TOAST
- Asking a Kubernetes cluster for a volume — covered in Kubernetes for Data Workloads, Series 2 · 05. In this course, Series 4 · 55 covers only how Ceph ends up being what answers that request
- Provisioning storage with infrastructure as code — covered in Terraform & Pulumi, Series 2 · 05
- Enterprise SAN arrays, fibre channel and vendor appliances — not covered. This track is storage a data team can obtain, run and reason about, and a six-figure appliance with a support contract is somebody else's job