Concept 9: Distributed File & Object Storage

System Design Bible — File 09 of 10 Difficulty: ★★★★☆ Prerequisites: Files 04, 06, 08, 05, plus the storage fundamentals in Section 0.

Table of Contents#

0Prerequisites — What You Must Understand First#

This file is about storing huge unstructured files (photos, video, backups) reliably across many machines. Three ideas set it up.

0.1A "blob" and why it's different from a database record#

A blob (Binary Large OBject) is a chunk of unstructured bytes stored and retrieved as a whole — a 4 MB photo, a 2 GB video, a 500 GB backup file. This is fundamentally different from the structured records databases handle (File 11): a database row (user_id, name, email) is small, has named typed fields, and you query and filter and join it. A blob has no queryable internal structure the storage system cares about — you PUT the whole thing and later GET the whole thing back; you don't ask it "find the part where the cat appears." This distinction drives the entire file: databases are optimized for many small structured records you query, while blob storage is optimized for fewer, much larger opaque objects you read/write whole. The classic rule you'll see — "store the blob in object storage, store its metadata and URL in the database" — exists precisely because the two need completely different systems. Hold it: a blob is a whole-read/whole-write lump of bytes; a record is a small queryable structured thing.

0.2Failure is the normal case, not the exception (and redundancy is the only defense)#

The single assumption that shapes all distributed storage: at scale, hardware fails constantly. A system with tens of thousands of disks will see multiple disks fail every single day — not as a rare disaster, but as a statistical certainty (a disk has, say, a ~1–2% annual failure rate; multiply by 50,000 disks and you're losing several a day). So a storage system that treats disk failure as an emergency is unusable; it must treat failure as routine and keep working through continuous hardware death without losing a single byte. The only defense against a disk losing data is to not have the only copy on that disk — i.e., redundancy: store the data in multiple places (multiple copies via replication, or split-with-parity via erasure coding, both in §6) across independent machines/racks/data centers, so any single failure (or several) loses nothing recoverable. Internalize this framing (it's literally the opening premise of Google's GFS paper): you are not building storage that avoids failure; you are building storage that expects failure and survives it via redundancy.

0.3Durability vs availability (two different promises)#

Two words that sound similar but mean different things, and mixing them up is a classic error. Durability = the probability your data is never lost — that once stored, it will still exist and be correct years later. Cloud object stores advertise "eleven nines" (99.999999999%) durability, meaning loss is astronomically unlikely, achieved by redundancy across many independent failure domains. Availability = the probability the data is **reachable right now — that a request to read it succeeds this instant. These are independent: data can be perfectly durable (safely stored, not lost) yet temporarily unavailable** (a network partition or service outage means you can't reach it for ten minutes, but it's still safely there and comes back). Object stores typically offer more nines of durability than availability, because losing data forever is far worse than a brief outage. Keep them separate: durable = will never be lost; available = can be read at this moment. (This connects directly to File 08's CAP — availability is the "A"; durability is about surviving failures.)

0.4What a disk physically is — HDD mechanics, SSD mechanics, and why sequential beats random#

Every design decision in this file is downstream of one fact: the device that actually holds your bytes is slow in a very specific, very lopsided way. You cannot reason about chunk sizes, the small-file problem, erasure coding, or storage tiers until you know why a disk behaves the way it does. So we start below the software entirely, at the physics.

The hard disk drive (HDD). An HDD is a stack of rigid metal platters coated in a magnetic film, spinning continuously at a fixed rate — 7,200 RPM in a desktop/nearline drive, 10,000 or 15,000 RPM in older enterprise drives, 5,400 RPM in laptop and high-capacity archive drives. Floating a fraction of a micron above each platter surface is a read/write head on the end of a pivoting actuator arm. The surface is divided into concentric circles called tracks, and each track is divided into sectors — the smallest physically addressable unit, historically 512 bytes, on modern drives 4,096 bytes (4 KB), a format called Advanced Format. To read the bytes at some location, two mechanical things must happen, and they are the entire story:

Add them: an average random access on a 7,200 RPM drive costs roughly 4 ms (transfer-independent seek, using the optimistic end) + 4.17 ms (rotation) ≈ 8–12 ms before a single useful byte arrives. Now invert that: a drive can perform about 1 second ÷ 10 ms = 100 random operations per second. That number — ~100–200 IOPS (IOPS = I/O Operations Per Second, the count of independent read or write requests a device can service each second, as distinct from how many bytes those requests move) — is the single most important constant in storage design. It has been ~100–200 for the entire history of the spinning disk.

Now contrast that with sequential access. Once the head is positioned, the platter keeps spinning and bytes stream under the head continuously with no further seeks and no further rotational waits. A modern 7,200 RPM nearline drive sustains 150–280 MB/s of sequential transfer. Do the comparison honestly, with 4 KB requests:

That is a 500× difference in delivered bandwidth from the same physical device, decided purely by whether the requests are adjacent. This is the "sequential beats random by orders of magnitude" law, and it is why GFS chose 64 MB chunks (§4.2), why the small-file problem exists (§4.9), why log-structured designs like Kafka (File 07) and Haystack (§12.1) win, and why archival systems happily use tape. When you see a design that turns random I/O into sequential I/O, you are looking at someone who understood this table.

The solid-state drive (SSD). An SSD has no moving parts. It stores bits as trapped electrical charge in NAND flash cells. Because there is no arm and no platter, there is no seek and no rotational latency, so random access costs almost the same as sequential access — a random 4 KB read on an NVMe SSD (NVMe = Non-Volatile Memory express, the modern protocol that talks to flash over PCIe lanes instead of pretending to be a disk on a SATA cable) completes in 50–100 microseconds, and the device sustains 500,000 to 1,000,000+ IOPS and 3–7 GB/s sequential on PCIe 4.0. A SATA SSD is capped by the SATA link at about 550 MB/s and ~90,000 IOPS. So flash is roughly 100× lower latency and 5,000× more IOPS than a spinning disk.

But flash has its own physics, and three terms fall out of it that you must know because they explain SSD pricing, endurance, and the weird failure behaviours of flash-backed storage:

Sub-Concept: Pages, erase blocks, and why flash cannot overwrite in place. NAND flash is organized into pages (the unit you can read or program/write, typically 4–16 KB) grouped into erase blocks (the unit you can erase, typically 128 pages to several thousand pages, so 256 KB to 4 MB+). The asymmetry is the whole problem: you can program a page from 1s to 0s, but you cannot rewrite a page that already holds data — you can only erase an entire block back to all-1s and then program its pages fresh. So "overwrite these 4 KB" is physically impossible as stated. The drive's internal firmware, the FTL (Flash Translation Layer), handles this by writing the new data to a different, already-erased page, updating an internal map from logical block address → physical page, and marking the old page invalid. Later, a background garbage collection process picks a block that is mostly invalid, copies the few still-valid pages out to a fresh block, and erases the old block to reclaim it.

Sub-Concept: Write amplification. That garbage-collection copying means the drive physically writes more bytes to NAND than your application asked it to write. The ratio is write amplification (WA): bytes written to NAND ÷ bytes written by the host. Worked example: you write a 4 KB page; garbage collection later has to relocate 60 still-valid pages from that block to free it; the true NAND cost of your 4 KB write was 4 KB + 240 KB = 244 KB, a write amplification of 61×. Real drives with decent over-provisioning and sequential-ish workloads run WA of 1.1–3×; small random writes on a nearly-full drive can push it to 10× or more. The consequences are practical: random-write-heavy workloads wear a drive out faster, and drive performance collapses when the drive is nearly full because there are few free blocks to absorb writes. The mitigation is to write sequentially and in large units — which is, again, the same lesson as the HDD table.

Sub-Concept: P/E cycles and wear levelling. Each erase block tolerates a finite number of program/erase (P/E) cycles before its insulating oxide degrades and it stops holding charge reliably: roughly 100,000 for SLC (1 bit per cell), 3,000 for MLC (2 bits), 1,000–3,000 for TLC (3 bits, the mainstream consumer/datacentre type), and ~200–1,000 for QLC (4 bits, used for read-heavy cheap storage). Because blocks wear out, the FTL performs wear levelling: it deliberately spreads writes across all blocks — including moving cold, never-updated data off a low-wear block so that block can absorb new writes — so no single block is destroyed while others sit unused. Endurance is sold as DWPD (Drive Writes Per Day) or TBW (Terabytes Written): a 4 TB drive rated 1 DWPD over 5 years tolerates 4 TB written per day for 5 years ≈ 7.3 PBW. What breaks: exceed the endurance and blocks are retired until the drive goes read-only or dies; you diagnose it with SMART attributes (Percentage Used/Media Wearout Indicator via smartctl -a /dev/nvme0). Tie-back: this is why write-heavy distributed storage is priced differently from read-heavy, and why erasure-coding repair traffic (§6.6) is not free — repair writes consume real drive lifetime.

The numbers you should be able to recite in an interview:

Notice the shape: as you go down the list, latency degrades by orders of magnitude while sequential bandwidth degrades only mildly and cost per GB collapses. That shape is the storage-tier hierarchy of §9. Hold this: capacity is cheap, bandwidth is moderate, and random operations are the scarce resource — so good distributed storage design is mostly the art of avoiding random I/O and per-object overhead.

0.5What a filesystem actually is — blocks, inodes, directory entries, and the per-file metadata tax#

A raw disk offers only "read sector 918,273" — an enormous flat array of numbered sectors with no names, no sizes, no ownership, no directories. A filesystem is the software layer (ext4, XFS, NTFS, APFS, ZFS, Btrfs) that builds the illusion of named, growable, permissioned files on top of that array. Understanding its data structures is mandatory here, because the cost of those structures is exactly what object storage was invented to escape.

A filesystem lays the disk out into four kinds of region:

Now the arithmetic that explains this entire file. Storing one file costs you, at minimum: one inode (256 B) + one directory entry (~20–60 B including the name) + directory index overhead + at least one 4 KB data block. Call the metadata cost roughly 300 bytes to 1 KB per file, plus the operational cost of the path lookups needed to reach it. Now imagine one billion small files: that is ~1 TB of pure metadata and — far worse — a billion index entries that must be searched, cached, backed up, and kept consistent. And every read of a small file costs at least one seek for the directory/inode and one for the data, so on a spinning disk, 2–3 seeks per file at ~100 IOPS means one disk serves roughly 30–50 small files per second no matter how fast its 200 MB/s streaming rate is.

That paragraph is the seed of three things you will meet later: the small-file problem in HDFS (§4.9), Facebook's Haystack design (§12.1) whose entire purpose is to delete per-file metadata cost, and object storage's decision to have no directory tree at all (§5.1). Hold this: a filesystem's per-file metadata and path-walk cost is negligible for a thousand big files and fatal for a billion small ones.

0.6POSIX semantics — the promises that are expensive to keep across a network#

POSIX (Portable Operating System Interface) is the IEEE standard that defines what a Unix-like filesystem must do. When someone says a distributed filesystem is "POSIX-compliant," they mean unmodified software — your text editor, your compiler, PostgreSQL, a legacy Fortran program — will work on it without knowing it is distributed. That compatibility is enormously valuable, and it is also the reason distributed filesystems are so much harder to scale than object stores. Here are the specific guarantees, what each one means, and precisely why each becomes expensive once the filesystem spans a thousand machines.

Hold this: object storage is best understood as "a filesystem with every expensive POSIX promise deliberately removed" — no rename, no partial writes, no hierarchy, no links, no locks — and the near-infinite scale of §5 is exactly what those removals buy.

0.7RAID — the single-machine ancestor, and why growing disks killed it#

Before data was spread across machines, it was spread across the disks inside one machine. RAID (Redundant Array of Independent Disks, from a 1988 Berkeley paper by Patterson, Gibson and Katz) is the set of schemes for doing that. You must know it for two reasons: interviewers ask, and erasure coding (§6) is RAID's idea generalized across a network — the arithmetic is literally the same.

First, the primitive every parity scheme is built on:

Sub-Concept: XOR parity. XOR (exclusive or, written ⊕) is the bitwise operation returning 1 when its two input bits differ and 0 when they match. It has the magic property that makes single-parity redundancy work: if P = A ⊕ B ⊕ C, then any one of the four values can be recovered from the other three, because A = P ⊕ B ⊕ C. Worked example with 4-bit values: A=1011, B=0110, C=1100. Then P = 1011 ⊕ 0110 = 1101, and 1101 ⊕ 1100 = 0001, so P=0001. Now lose B entirely. Recover it: P ⊕ A ⊕ C = 0001 ⊕ 1011 ⊕ 1100 = 0110 — exactly B. Cost: one extra disk's worth of space protects an arbitrary number of data disks against one loss. Limit: XOR gives you exactly one equation, so it can solve for exactly one unknown — two simultaneous losses are unrecoverable. Getting past that limit requires the Galois-field mathematics of Reed–Solomon (§6.4).

The RAID levels, each with its mechanism, its arithmetic, and its failure mode:

Now the punchline — why RAID stopped being enough, which is the direct motivation for everything after §6. Rebuild time scales with disk capacity, but rebuild speed is bounded by disk bandwidth, and capacity has grown far faster than bandwidth. Work it:

And there is a second, sharper problem: the URE (Unrecoverable Read Error) rate. Drive datasheets quote roughly 1 URE per 10¹⁴ bits read for consumer drives and 10¹⁵ for enterprise ones. 10¹⁴ bits = 12.5 TB. To rebuild one failed 20 TB drive in a 6-disk RAID 5, you must read the other five drives completely: 5 × 20 TB = 100 TB = 8 × 10¹⁴ bits. At 1 URE per 10¹⁴ bits, you should expect around eight unrecoverable read errors during a single rebuild — and in RAID 5, with no remaining redundancy, each one is unrecoverable data loss. RAID 5 on large modern drives is statistically expected to fail its own rebuild. That is not a hypothetical; it is why the industry moved to RAID 6, and then past RAID entirely.

RAID has three further limits no level fixes: it protects against disk failure inside one chassis, so a failed motherboard, a power event, a fire, or a rack switch takes everything; classic RAID has no checksums, so if a disk returns wrong-but-plausible bytes, RAID happily serves them (§7); and rebuild reads from a small fixed set of disks, so rebuild bandwidth cannot be increased by adding hardware.

Distributed erasure coding fixes all three at once: fragments are spread across many independent machines, racks, and datacentres; every fragment carries a checksum; and because a failed drive's fragments belonged to thousands of different stripes scattered over hundreds of surviving drives, the rebuild reads from hundreds of disks in parallel — declustered repair — which turns a 4.6-day single-drive rebuild into a ~20-minute cluster-wide one. Hold this: RAID is erasure coding confined to one box; the rebuild-time and URE arithmetic on 20 TB drives is what forced the idea out of the box and onto the network.

0.8Durability as arithmetic — what "eleven nines" actually means and how the math works#

"Eleven nines of durability" is the most repeated and least understood phrase in storage. Here is what it means numerically and how the number is produced, because an interviewer asking "how would you get eleven nines?" is asking for this arithmetic.

What the phrase asserts. 99.999999999% annual durability = a 10⁻¹¹ probability that a given object is lost in a given year. Read it as an expectation: **if you store 10,000,000 (10⁷) objects, the expected number lost per year is 10⁷ × 10⁻¹¹ = 10⁻⁴ — one ten-thousandth of an object per year, i.e., you would expect to wait 10,000 years to lose one object. Equivalently, storing 100 billion objects (10¹¹) you would expect to lose about one object per year**. Note the number is per object per year, which is why it sounds absurd and is nonetheless a meaningful engineering target at exabyte scale.

How independent replicas multiply. The core move is that independent probabilities multiply, so each redundant copy adds orders of magnitude, not increments. Set up the model with real values:

Now, for a chunk stored on 3 independent disks, data is lost only if all three die and the second and third deaths land inside the repair window opened by the first:

P(first copy's disk fails this year)                    = 0.02
P(a specific second disk fails within the 1h window)    = 0.02 × 1.14e-4  = 2.28e-6
P(the third disk fails within the window too)           = 0.02 × 1.14e-4  = 2.28e-6
                                                          ------------------------
P(all three lost in one year) ≈ 0.02 × 2.28e-6 × 2.28e-6 ≈ 1.0e-13

1.0 × 10⁻¹³ is twelve-and-a-half nines — comfortably past eleven. Run the same arithmetic for 2 copies and you get 0.02 × 2.28e-6 ≈ 4.6 × 10⁻⁸, about seven nines — five orders of magnitude worse. That single comparison is the whole argument for R=3 rather than R=2, and you should be able to produce it on a whiteboard.

Two structural lessons fall out of the formula, and they matter more than the exact digits:

Sub-Concept: Reading "nines" fluently. The count of nines is just the negative base-10 logarithm of the failure probability. 99.9% = 10⁻³ = three nines; 99.99% = four nines; eleven nines = 10⁻¹¹. It is worth memorizing the availability translation too, since availability nines are quoted as downtime: 99% = 3.65 days/year down; 99.9% = 8.77 hours; 99.99% = 52.6 minutes; 99.999% = 5.26 minutes; 99.9999% = 31.6 seconds. S3 Standard advertises 99.999999999% durability but only 99.99% availability — about 52 minutes of allowed unreachability per year — which is the numerical form of the durability ≠ availability point in §0.3.

1Architectural Definition & "The Why"#

1.1The Plain Definition#

Distributed file/object storage is a system that stores files or objects (blobs) — images, videos, backups, logs, ML datasets, documents — across many machines, presenting them as one giant, reliable storage pool that can hold petabytes to exabytes, survive constant hardware failure, and serve massive read/write throughput. Examples: Google File System (GFS), HDFS, Amazon S3, Ceph, Azure Blob, MinIO.

This is distinct from the databases of Files 04/08. Databases store structured records you query (rows, key-values) with transactions and indexes. This file is about storing large unstructured blobs you read/write whole — where the challenges are capacity, durability, and throughput, not complex queries.

1.2"The Why" — Blobs Don't Fit the Database Model, and Disks Die#

Two forces created distributed blob storage:

So you need a system that: spreads data across thousands of cheap commodity disks (capacity + throughput via parallelism), automatically keeps redundant copies so no failure loses data (durability), and heals itself when drives die — all while presenting a simple "put/get a file" interface.

1.3What It Solves#

2The Recursive Sub-Concept Tree#

2.1Sub-Concept: The Control Plane vs. Data Plane Split#

The single most important idea in distributed storage: separate metadata from data.

Clients ask the metadata server "where is file X?", get back a list of storage-node locations, then read/write the bytes directly from/to the storage nodes — the bulk data never flows through the metadata server (which would be a bottleneck). This control/data-plane separation is the architectural backbone of GFS, HDFS, Ceph, and most blob stores.

2.2Sub-Concept: Chunking / Sharding of Files#

A large file is split into fixed-size chunks (GFS: 64 MB; HDFS block: 128 MB) or variable blocks. Each chunk is stored (and replicated) independently across nodes. Benefits: a huge file spreads across many disks (parallel throughput), no file is limited by one disk's size, and only affected chunks move during rebalancing. This is File 04's partitioning applied to file bytes. Large chunk sizes reduce metadata volume (fewer chunks to track) and favor sequential/streaming access.

2.3Sub-Concept: Replication Factor#

Each chunk is stored on R nodes (typically R=3). If a node dies, the chunk still exists on R−1 others, and the system re-replicates it elsewhere to restore R. Rack/AZ awareness (place the 3 copies on different racks/availability zones) ensures a whole-rack or whole-AZ failure doesn't take all copies. (Deep dive + the erasure-coding alternative in Section 6.)

2.4Sub-Concept: Durability vs. Availability (they're different!)#

2.5Sub-Concept: Immutability & Versioning (object storage)#

Object stores typically treat objects as immutable: you don't edit an object in place; you overwrite it with a new version (a whole new PUT). This simplifies caching, replication, and consistency enormously (no partial-update coordination). Versioning keeps old versions on overwrite/delete (protection against accidental loss). Immutability is why object storage pairs so cleanly with CDNs and content-hashed URLs (File 05).

2.6Sub-Concept: Eventual vs. Strong Consistency for Storage#

Historically S3 was eventually consistent (a just-written object might briefly 404, or a read might return an old version). As of Dec 2020, S3 provides strong read-after-write consistency for all operations. Distributed stores must decide: after a PUT, is a subsequent GET guaranteed to see it? (Ties directly to File 08.) The full treatment — what changed in December 2020, why LIST is the hardest operation to make consistent, and what strong consistency still does not give you — is §8.

2.7Sub-Concept: Failure Domains and Rack/AZ-Aware Placement#

Plain definition. A failure domain (or fault domain) is a set of components that can plausibly fail together because they share some physical or logical dependency. Two disks in the same server share a motherboard, a power supply, and a NIC — one failure domain. Two servers in the same rack share a top-of-rack switch and a power distribution unit — a bigger failure domain. Two racks in the same building share a UPS, a cooling plant, and a fibre entrance — bigger still.

Why it exists. Because of the independence problem in §0.8: the durability arithmetic that turns 3 copies into twelve nines is only valid if the three failures are independent. Placing all three replicas of a chunk on three disks in the same rack does not give you three independent 2% events; it gives you approximately one correlated event — the rack's power or switch dying — at maybe 0.5% per year. You did not buy twelve nines, you bought about two and a half, while paying for three copies. Failure-domain awareness is the mechanism that makes the multiplication in §0.8 legitimate.

Where it sits. In the placement policy of the metadata/control plane (§2.1): the component that decides, for each new chunk, which storage nodes will hold its replicas or fragments. Every serious system has one, and every one of them is configured with a topology description.

How it works, step by step, with concrete values. The cluster is described as a tree of location labels: region → availability zone → datacentre → row → rack → node → disk. HDFS calls this rack awareness and gets the mapping from a script (net.topology.script.file.name) or a static table (net.topology.table.file.name) that turns an IP into a path string like /dc1/rack17. Ceph calls it the CRUSH map (§15.2) and encodes the tree as buckets of types root, datacenter, rack, host, osd. The placement rule is then a constraint over that tree. HDFS's default rule for replication factor 3 is worth memorizing exactly, because it is a beautifully argued compromise:

Replica 1 -> the local node (the node where the writing client runs; if the client
             is outside the cluster, a randomly chosen node)   -> zero network cost
Replica 2 -> a node on a DIFFERENT rack                        -> survives a whole-rack loss
Replica 3 -> a DIFFERENT node on the SAME rack as replica 2    -> cheap: intra-rack copy

Why not put all three on three different racks? Because cross-rack bandwidth is the scarce resource in a datacentre — a top-of-rack switch has, say, 48 × 25 Gbps of downlink to its servers but only 4 × 100 Gbps of uplink to the spine, an oversubscription ratio of 3:1 — and a third cross-rack copy would spend that scarce uplink bandwidth to protect against an event (two whole racks failing simultaneously) that is far rarer than the single-rack event replica 2 already covers. So the rule buys the big win (rack independence) for two rack-crossings' worth of traffic, not three. That trade — durability against cross-rack bandwidth — is the answer to "why is HDFS's default placement policy shaped like that?", and it is a genuine senior-level interview question.

In a cloud object store the same idea appears one level up: S3 Standard places redundancy across at least three Availability Zones, where an AZ is one or more discrete datacentres with independent power, cooling and networking, typically kilometres apart but close enough (usually under ~100 km) that inter-AZ round-trip latency stays around 1–2 ms so synchronous replication remains practical. S3 One Zone-IA (§9.2) deliberately gives that up — same eleven nines against disk failure, but a lost AZ loses the data — in exchange for about 20% lower price.

What breaks. Three classic failures. (1) A wrong or missing topology map: if the rack script returns /default-rack for every node — which is what happens when the script is missing or errors — the system believes every node is in one rack, places replicas arbitrarily, and reports full health while being one PDU away from total loss. You diagnose it with hdfs dfsadmin -printTopology, which shows every node's resolved rack path. (2) Domain collapse after a topology change: you migrate hosts into fewer racks and placement silently stops satisfying the rule for already-written data; the fix is a rebalance pass that re-verifies placement, not just capacity. (3) Correlated domains you did not model: same drive batch, same firmware version, same software daemon, same cooling loop. Google and Facebook both publish incidents where the modelled domains were fine and an unmodelled one (a bad firmware rollout) hit everything at once.

Tie-back. Every durability number in this file — the eleven nines of §0.8, the "survives any 3 losses" of §6.5 — is a claim about independent failures, and failure-domain-aware placement is the only thing that makes the word "independent" true.

2.8Sub-Concept: The Lease (a lock with an expiry, the fix for "who is in charge?")#

Plain definition. A lease is a lock granted for a bounded period of time: the holder has exclusive authority over something until a stated deadline, after which the authority evaporates automatically whether or not the holder is still alive or reachable.

Why it exists. Because in a distributed system you cannot distinguish "the node holding the lock has crashed" from "the node holding the lock is fine but the network to it is broken" (this is the impossibility that File 08 and File 18 are built around). A plain lock is therefore unsafe: if you never revoke it, one crashed node blocks progress forever; if you revoke it on suspicion, you may create two simultaneous holders — a split brain — and two writers ordering mutations differently on the same data means the replicas silently diverge. A lease resolves the dilemma by making expiry a property of time rather than of belief: the grantor promises not to reassign before the deadline, and the holder promises to stop acting after it. Both sides can enforce their half locally, with no message.

Where it sits. In GFS, on each chunk: the master grants a 60-second lease naming one replica the primary for that chunk (§4.4). In HDFS, on each open file: the NameNode grants a write lease to the single client that opened it for writing, enforcing HDFS's single-writer model, with a soft limit of 60 seconds (after which another client may preempt if the holder has stopped renewing) and a hard limit of 60 minutes (after which the NameNode forcibly recovers the file's last block and closes it). In Chubby/ZooKeeper (File 08), it appears as an ephemeral node whose lifetime is tied to a heartbeated session.

How it works, step by step, with concrete numbers. Take GFS. At t=0 the master grants chunk C's lease to replica DN7, deadline t=60s, and records it in the master's memory. DN7 orders all mutations to C for that period. If DN7 is still writing at t=45s, it piggybacks a renewal request on its regular heartbeat and the master extends the deadline to t=105s — renewals are cheap because they ride existing traffic. Now suppose DN7's machine dies at t=30s. The master notices missed heartbeats but does not immediately reassign; it waits until t=60s, the original deadline, because until then DN7 might be alive-but-partitioned and still ordering writes. At t=60s+ε the lease has expired by the clock, DN7 (if alive) has stopped accepting writes because it too can see its own deadline passed, and the master safely grants a new lease to DN12. The window of unavailability for that one chunk is at most the lease duration — 60 seconds — which is precisely the cost you pay for the safety of never having two primaries.

The exact parameters and what they mean. The lease duration is a direct trade: shorter leases mean faster failover (less unavailability after a primary dies) but more renewal traffic and more sensitivity to clock skew and transient network blips, which can cause a healthy primary to lose its lease and cause spurious churn. Longer leases mean less overhead and more stability but a longer stall after a real failure. Sixty seconds is the empirical sweet spot for chunk-granularity leases. HDFS's split of soft (60 s) and hard (60 min) limits is the same trade applied twice: preempt a possibly dead writer quickly, but only forcibly destroy its write after an hour when it is certainly gone.

The clock assumption, and what breaks. A lease is only safe if the grantor's and holder's clocks do not drift apart by more than the safety margin. If the holder's clock runs slow, it may believe it still holds a lease the master has already reassigned — two primaries, divergent replicas, silent corruption. Mitigations: keep clocks disciplined with NTP; have the holder expire early (treat the lease as shorter than granted by the maximum assumed skew, e.g., act for only 50 s of a 60 s lease); and, best of all, add a fencing token — a monotonically increasing number issued with each lease grant, which storage nodes record and use to reject any write carrying a lower token than the highest they have seen. Fencing makes safety independent of clocks entirely, and naming it is a strong interview signal.

Tie-back. The lease is what lets GFS give a consistent write order per chunk without running a consensus round for every write — the expensive coordination happens once per minute per chunk instead of once per mutation. That is the performance trick at the heart of §4.4.

2.9Sub-Concept: Checksums and the Hash Functions Behind Them#

Plain definition. A checksum is a small fixed-size value computed from a block of data such that if the data changes, the value almost certainly changes too. Store it alongside the data; recompute it on read; if the two disagree, the data is corrupt.

Why it exists. Because disks, controllers, cables, memory and networks all corrupt data silently — they return wrong bytes with no error indication (§7.2). Without a checksum, the storage system cannot tell good data from bad, so it faithfully serves corruption and, worse, faithfully replicates it.

Where it sits. At several independent layers, and knowing which is which matters: the drive's own per-sector ECC (below the OS, invisible to you); the filesystem's block checksums if it has them (ZFS and Btrfs do; ext4 checksums only metadata by default); the distributed storage system's own per-chunk or per-block checksum (HDFS stores a CRC-32C for every 512 bytes of data by default — dfs.bytes-per-checksum — in a companion .meta file beside each block replica); and the client-supplied end-to-end checksum carried in an HTTP header (S3's Content-MD5, or the newer x-amz-checksum-crc32c / -crc32 / -sha1 / -sha256 family).

How it works, and the algorithms individually.

Granularity is a real design parameter. Checksum a whole 128 MB block with one value and you must read all 128 MB to verify anything and you learn only that something is wrong. Checksum every 512 bytes and you can verify a small range read cheaply, and you learn where the corruption is — at the cost of 128 MB ÷ 512 B × 4 bytes = 1 MB of checksum data per block, a 0.78% space overhead. HDFS chose 512 B; many object stores checksum per part or per erasure fragment. The rule: choose the granularity to match your smallest expected read.

What breaks. (1) If the checksum is computed on the server after receiving the data, it protects the disk but not the network or the client's RAM — corruption that happened in flight becomes a perfectly-checksummed corrupt object. That is exactly why end-to-end, client-computed checksums exist (§7.3). (2) If checksums are stored on the same disk as the data with no redundancy, a failure that eats both leaves you unable to tell good from bad. (3) A checksum detects but does not repair; repair requires redundancy, which is why §7 and §6 are inseparable.

Tie-back. Checksums are what convert a silent corruption into a loud one, and only loud failures can be repaired by the replication and erasure-coding machinery of §6.

2.10Sub-Concept: The Namespace Vocabulary — Bucket, Key, Prefix, Delimiter, Object#

You will meet these five words on every page from here on, so pin them down now. A bucket is the top-level container and the unit of ownership, policy, region, and billing; its name is part of the URL and, in S3's original scheme, globally unique across all AWS customers (which is why test and backup have been taken since 2007). An object is the stored thing: a blob of bytes plus system metadata (size, last-modified, ETag, storage class) plus optional user metadata. A key is the object's full name inside the bucket — a single UTF-8 string up to 1,024 bytes, e.g. photos/2024/06/cat.jpg. There is no directory structure: that key is one flat string that merely happens to contain slashes. A prefix is any leading substring of a key that you use to select a group of objects (photos/2024/), and a delimiter is a character (conventionally /) that you pass to a LIST call to make the API roll up everything after the next occurrence of that character into a synthetic group, which is the trick that fakes folders (§5.2). Internalize the sentence: a bucket is a namespace, a key is a string, a prefix is a string comparison, and a folder is a rendering convention.

2.11Sub-Concept: The Three Amplifications (write, read, repair)#

Storage engineers describe most inefficiencies as an amplification — a ratio of work actually done to work logically requested — and you should be able to use all three terms precisely, because they are the vocabulary the rest of this file argues in.

3Three Storage Abstractions: Block, File, Object#

A crucial interview taxonomy — know all three, their interfaces, and when to use each.

3.1Block Storage#

Model: Exposes raw fixed-size blocks — like a virtual hard disk. No concept of files — just numbered blocks the OS/filesystem sits on top of. (The classic enterprise form is a SAN — Storage Area Network: a dedicated network of storage arrays that serves raw block volumes to servers over a fast fabric, so a server's "disk" is actually across the network but behaves exactly like a local drive.)

3.2File Storage#

Model: A hierarchical filesystem — directories, files, paths — shared over the network. (The appliance form is a NAS — Network Attached Storage: a box that serves files over the network, in contrast to a SAN which serves raw blocks; NAS speaks "open this file," SAN speaks "read block 4096.")

3.3Object Storage#

Model: A flat namespace of objects (blob + metadata + unique key), accessed via HTTP API (not a mounted filesystem). No real directories — the "folders" in S3 are just key prefixes (photos/2024/cat.jpg is one flat key with slashes).

3.4The Decision Table#

Interview line: "Databases and VMs want block; legacy apps that need a mounted filesystem want file; anything you read/write as a whole blob and serve over HTTP wants object."

The table above is the summary. Because "block vs file vs object" is one of the two or three most-asked storage questions in a system-design interview, and because a table is not an explanation, the next four sub-sections give each paradigm the full treatment: the actual mechanism, the exact API, why its design makes it good at what it is good at, where it breaks, and the decisive conditions for choosing it.

3.5Block Storage in Depth — mechanism, API, pros, cons, when to use#

Mechanism. A block device is presented to the operating system as an addressable array of fixed-size blocks — conceptually block[0], block[1], … block[N-1], each 512 bytes or 4 KB. The addressing scheme is LBA (Logical Block Addressing): you name a block by its integer index, and the device (or the network protocol pretending to be a device) translates that to physical location. There is no notion of a file, a name, an owner, or a size limit per item; there is only "read 8 blocks starting at LBA 1,048,576." Everything above that — files, directories, permissions — is built by a filesystem that the OS runs on top of the block device (§0.5), or by a database that manages the raw blocks itself.

The API, named precisely. At the OS level it is the block-device interface: read()/write()/pread()/pwrite() against a device node such as /dev/nvme0n1 or /dev/xvdf, plus fsync/fdatasync for durability barriers and TRIM/DISCARD to tell an SSD a range is no longer in use. Across a network the same interface is carried by a storage protocol: iSCSI (SCSI commands tunnelled over TCP/IP, the cheap and ubiquitous option), Fibre Channel (SCSI over a dedicated optical fabric with its own switches and HBAs — fast, deterministic, expensive, the classic SAN), NVMe-oF (NVMe over Fabrics) (the modern one, carrying NVMe commands over RDMA, TCP or Fibre Channel with far lower protocol overhead, and the reason network-attached flash can now approach local-flash latency), or a cloud provider's proprietary equivalent (AWS EBS's protocol between the hypervisor and its storage fleet).

Why its design makes it fast. The interface is thin. There is no name to resolve, no metadata to consult, no HTTP header to parse, no authentication per operation — just an integer and a length. And because the client owns the filesystem, it can cache aggressively and lay data out to suit itself. This is why block is the only paradigm on which you can sensibly run a database: a B-tree page update is a 8 KB in-place write at a computed offset, exactly what a block device does natively and exactly what an object store forbids.

Pros, with the reason for each. Lowest latency — sub-millisecond, because there is no layer between you and the media beyond a thin protocol. In-place mutability at arbitrary offsets — the essential property for databases, VM disks, and journals. Real filesystem semantics — because you install a real filesystem on it, you get every POSIX guarantee of §0.6 for free, and they are cheap because there is exactly one machine mounting the volume. Predictable, provisionable performance — cloud block volumes let you buy a specific IOPS and MB/s number (§15.5).

Cons, with the failure mode. Single-attach — a block volume is normally attached to one instance at a time, because two machines running independent filesystems over the same blocks will destroy the filesystem within seconds: each caches metadata the other is mutating, with no coordination. (Multi-attach exists — AWS EBS Multi-Attach, shared LUNs — but it is only safe if you run a cluster-aware filesystem such as GFS2 or OCFS2, or a database that does its own distributed locking, e.g. Oracle RAC. Attaching an ext4 volume to two hosts is a data-loss bug, not a configuration.) Capacity is bounded by the volume — you provision 16 TB and you have 16 TB; growing means an explicit resize plus a filesystem grow, not an infinite namespace. You pay for provisioned space, not used space — an empty 1 TB EBS volume costs the same as a full one, unlike object storage which bills bytes actually stored. No built-in HTTP access, no global namespace, no CDN story. Durability is narrower — an EBS volume is replicated within a single Availability Zone, giving about 99.999% (five nines) annual durability for gp3, which is a million times weaker than S3's eleven nines; block storage expects you to take snapshots (which land in object storage) for real protection.

When to use it — the decisive conditions. Choose block when any of these is true: you are running a database or any system that manages its own on-disk structures; you need a bootable root volume for a VM; your workload does small random in-place updates; you need sub-millisecond latency; or you have software that simply requires a local disk and cannot be changed. Do not choose block for data that many machines must read simultaneously, for data that must be reachable over HTTP, or for anything petabyte-scale — those are the other two paradigms' jobs.

3.6File Storage in Depth — mechanism, API, pros, cons, when to use#

Mechanism. A file server exports a hierarchical namespace — a tree of directories containing files — and clients mount it so that the tree appears at a path in their own filesystem (/mnt/shared). When a process on the client opens /mnt/shared/reports/q3.csv, the client's kernel recognises the path as belonging to a network filesystem and turns each POSIX call into a remote procedure call to the server. The abstraction is exactly a filesystem; the only difference is that the disk is on the other side of a network, and that many clients share it concurrently — which is where all the difficulty lives.

The API and protocols, named individually. The client-visible API is POSIX (§0.6): open, read, write, lseek, close, stat, readdir, rename, link, chmod, flock. The wire protocols are:

Why its design makes it convenient. Because it is a filesystem, every piece of existing software works unmodified, and multiple machines can genuinely share and mutate the same data with a coherent view — the thing neither block (single-attach) nor object (immutable, no partial writes) can offer.

Pros with reasons. Zero application change — the killer feature; you cannot overstate how much software assumes a filesystem. Concurrent shared read/write across many clients with real locking, which is what makes shared home directories, build artifacts, and CMS media directories work. Partial in-place updates — unlike object storage. Hierarchical organisation that humans and tools navigate naturally.

Cons with the mechanism of failure. Metadata is the scaling wall. Every open walks the path (§0.5) and every stat is a round trip, so a workload of many small files becomes a storm of tiny RPCs — a build system doing 200,000 stat calls over NFS at 0.5 ms each spends 100 seconds doing nothing but path lookups. Consistency across clients is expensive: to let client A see client B's write, the server must either invalidate A's cache (requiring server-tracked state and callbacks) or force A to revalidate constantly (requiring round trips). Locks have a liveness problem: a client holding a POSIX lock that loses power leaves the lock held until a lease expires, stalling everyone else. Scale ceiling: shared filesystems typically top out at the low petabytes with a single namespace, well short of object storage's effectively unbounded keyspace. Cost: managed NFS is dramatically more expensive per GB than object storage — AWS EFS Standard is roughly $0.30/GB-month against S3 Standard's $0.023/GB-month, a 13× premium you are paying for POSIX.

When to use it — the decisive conditions. Choose file storage when the software cannot be changed and demands a filesystem; when several machines must read and write the same mutable data with coordination (shared scratch space, CI artifacts, a legacy CMS's upload directory, an HPC job's working set); or when humans need to browse the data with ordinary tools. Do not choose it as the storage layer for a new internet-facing application's user uploads — that is object storage's job and file storage will cost you an order of magnitude more for a semantic you did not need.

3.7Object Storage in Depth — mechanism, API, pros, cons, when to use#

Mechanism. An object store is, at its core, a giant distributed key-value map from a string key to a blob plus metadata, accessed over HTTP, in which values are immutable and there is no hierarchy. That is the entire model, and its power comes from what it removed: no directory tree to lock or traverse, no inodes, no rename, no partial writes, no locks, no link counts. Every one of those removals eliminates a piece of cross-machine coordination, and what is left — "look up a key in a sharded index, then stream bytes from the nodes it names" — shards essentially perfectly (Files 04 and 06). §5 dissects the internals; §5.2–5.7 dissect the API operation by operation.

Why its design makes it scale. Two reasons, and you should be able to state both. First, the flat keyspace means every operation is a point lookup or a range scan over a sorted index, both of which are trivially partitionable across thousands of machines — whereas a directory tree forces a hierarchical lock and consistency structure whose upper nodes are inherently hot. Second, immutability means an object never needs to be updated in place, so replicas can never disagree about the order of concurrent mutations — the hardest problem in distributed data (File 08) is defined out of existence. The price of both is exactly the POSIX semantics of §0.6, and that is a deliberate, correct trade, not an oversight.

Pros with reasons. Effectively unbounded scale (trillions of objects, exabytes) for the reasons above. Extreme durability at low cost — eleven nines at $0.023/GB-month, because erasure coding across failure domains (§6) is cheap when you never have to modify a stripe. HTTP-native, so it is a perfect CDN origin (File 05), directly browser-reachable via presigned URLs (§10.4), and usable from any language with an HTTP client. Rich per-object metadata and tags that you can drive lifecycle and access policy from. Pay-for-what-you-store rather than pay-for-provisioned. Built-in versioning, replication, lifecycle and encryption as configuration rather than code.

Cons with the mechanism. Latency is milliseconds, not microseconds — first-byte latency for a GET is typically 20–100 ms even within the same region, because every request is an authenticated HTTP request that consults a distributed index before touching data. That is 100–1,000× a local NVMe read (§0.4) and it makes object storage unusable as a database's primary storage without an aggressive caching layer. No partial writes — changing one byte of a 5 GB object means uploading 5 GB (or using the copy-part trick of §5.5). LIST is slow and expensive — it is a paginated range scan at ~1,000 keys per request, so enumerating a bucket with 100 million objects is 100,000 requests; you should keep an external index (a database table) if you need to query your objects. No cross-object transactions (§8.5) — you cannot atomically update two objects, which is exactly the gap table formats (§12.3) fill. Request and egress charges can dominate the bill (§9.7) in ways that surprise people who budgeted only for storage. Eventual consistency historically, and still for some operations in some providers (§8).

When to use it — the decisive conditions. Choose object storage when the unit of work is a whole blob written once and read many times: user uploads, images, video, audio, PDFs, static site assets, backups and snapshots, database dumps, log archives, ML training datasets and model checkpoints, data-lake tables (§12.3), and anything that will be served through a CDN. Do not choose it for a database's live data files, for small-random-update workloads, for anything needing sub-millisecond latency, or as a POSIX substitute via a FUSE mount (s3fs, goofys, Mountpoint for S3) unless you have verified your access pattern is large sequential reads — those tools translate POSIX calls into HTTP requests and turn one innocent ls -lR into tens of thousands of billable LIST/HEAD calls.

3.8The Decision Rule — pick by workload, not by fashion#

Say this out loud in an interview, in this order:

The anti-patterns to name explicitly: putting blobs in the database (§1.2) — it destroys the buffer pool and makes backups enormous; using object storage as a POSIX filesystem through a FUSE shim and then wondering why ls takes four minutes and the bill has a five-figure request line; using EFS/NFS to store 200 million static images because "it's a filesystem and we know filesystems," paying 13× the price for locking semantics no one uses; and putting a transactional database's data directory on NFS, where fsync semantics and lock recovery differ from local disk in ways that produce corruption after a client crash.

4Deep-Dive Mechanics: The GFS/HDFS Architecture#

Google File System (GFS) and its open-source twin HDFS are the canonical designs. Master this flow — it recurs everywhere.

4.1The Three Components#

Three details about those components decide the whole design and are worth stating precisely, because they are what an interviewer probes.

The master holds three maps, and only two of them are persistent. (1) The namespace: the full path→file mapping, stored as a lookup table of full pathnames (not a real tree of inodes — GFS deliberately has no per-directory data structure and no inode-like objects, which is why it has no hard links and cheap-to-fake directories). (2) The file→chunk mapping: for each file, the ordered list of 64-bit chunk handles that compose it. (3) The chunk→location mapping: for each chunk handle, which chunkservers currently hold a replica. Maps (1) and (2) are persisted to the operation log (§4.6) because they cannot be reconstructed from anywhere else. Map (3) is not persisted at all — at startup, and periodically thereafter, the master simply asks every chunkserver what it has. That is a genuinely elegant decision, and the reasoning is: the chunkserver is the authority on what is on its own disks, so persisting the master's belief about it would only create a second source of truth that could drift and would need reconciliation after every disk failure, every corruption, and every out-of-band move. Ask, don't remember.

The chunkserver stores each chunk as an ordinary Linux file on an ordinary local filesystem (ext4/XFS), named by its chunk handle, with a companion checksum file (§2.9). This is why the design was cheap to build and cheap to operate: the chunkserver is a few thousand lines of code over a filesystem that already works, not a bespoke on-disk format. It also means the storage layer inherits the local filesystem's own guarantees and its own limits.

The client is a library, not a daemon. In both GFS and HDFS the "client" is a linked-in library inside the application process. It caches chunk locations (with a timeout, typically minutes), computes which chunk an offset falls in by simple division, and does its own failover between replicas. The consequence: clients are semi-trusted participants in the protocol, and a buggy client can hammer the master. HDFS's answer is the same library plus server-side rate limits; the modern object-storage answer (§5) is to make the client a plain HTTP caller with no protocol knowledge at all.

4.2Why the Chunk Is 64–128 MB (the arithmetic that justifies the number)#

This is the single most-asked "why" in the GFS/HDFS design, and the answer is two independent arithmetic arguments that happen to point the same way. A typical local filesystem uses a 4 KB block (§0.5); GFS chose 64 MB and HDFS defaults to 128 MB (dfs.blocksize, and clusters that hold very large files often raise it to 256 MB). That is a factor of 16,000–32,000 larger than a filesystem block. Here is why.

Argument 1 — metadata volume. The master keeps all metadata in RAM, because a metadata operation must be fast (microseconds, not a disk seek) and because there are a lot of them. GFS reports about 64 bytes of metadata per chunk (the handle, version, replica list, reference count), plus roughly 64 bytes per file for the namespace entry with prefix compression. Now hold the cluster capacity fixed at 10 petabytes and vary the chunk size:

That table is the whole reason the single-master design was viable at all. With 64 MB chunks, a single machine's RAM can index a petabyte-scale cluster; with 1 MB chunks it cannot, and you would have been forced into a distributed metadata layer in 2003 instead of 2010 (which is exactly the transition Colossus later made, §16.2).

Argument 2 — seek amortization. From §0.4, a spinning disk costs ~10 ms to start reading anywhere and then streams at ~200 MB/s. Define efficiency = transfer time ÷ (seek time + transfer time):

The curve flattens hard somewhere around 64–128 MB: going from 4 KB to 64 MB buys you a 485× improvement in disk efficiency, and going from 64 MB to 1 GB buys you only another 1.5%. The chunk size is chosen at the knee of that curve — large enough that the seek is amortized to irrelevance, not so large that you pay the costs in argument 3.

Argument 3 — why not make it 10 GB, then? Three costs grow with chunk size. (a) Hot spots: a chunk is the unit of replication and therefore the unit of contention. If a small popular file fits in one chunk, all reads of it hit the same 3 chunkservers. GFS's paper describes exactly this happening with an executable distributed to hundreds of machines at once; the fixes were to raise the replication factor for that file and to stagger client start times. Bigger chunks make this worse because more of the file's traffic concentrates. (b) Internal fragmentation / wasted space: GFS uses lazy space allocation (a chunk file grows only as data is written) which mitigates this, but a system that pre-allocates would waste up to one chunk per file. (c) Recovery and rebalancing granularity: the unit you must copy to repair or rebalance is one chunk, so 10 GB chunks make every repair a 10 GB transfer and make load levelling coarse and lumpy.

**Also note the third-order effect, which is the client's benefit:** a large chunk means a client doing a long sequential read holds a single TCP connection to a single chunkserver for hundreds of milliseconds, and it means one metadata request from the master covers a huge span of file, so client-side location caching is extremely effective. GFS's paper explicitly lists reduced client–master interaction as a motivation.

Tie-back and the interview line: "64–128 MB because the master must hold all metadata in RAM — at 64 bytes per chunk, 10 PB is 10 GB of RAM at 64 MB chunks and 160 TB at 4 KB — and because a 10 ms seek is 0.2% efficient on a 4 KB read but 97% efficient on a 64 MB read. It is not larger because the chunk is also the unit of hot-spotting and repair."

4.3The Read Path#

1. Client -> Master: "give me chunk locations for file X, offset O"
2. Master -> Client: chunk handle + list of chunkservers holding replicas (e.g., [DN7, DN12, DN30])
3. Client -> nearest/available chunkserver (e.g., DN7): "send chunk"
4. DN7 -> Client: streams the bytes directly.
   (Bulk data NEVER passes through the master -> master isn't a bandwidth bottleneck)
   Client caches the chunk-location metadata to avoid re-asking the master for subsequent reads.

Now the same path drawn with the actual machines, message sizes, and timings, so you can see where the bytes are at every moment. The client wants bytes [70,000,000 … 70,004,095] — a 4 KB read at offset 70 MB — of a 1 GB file /data/big.log on a 128 MB-block HDFS cluster:

                          ┌──────────────────────────────────────┐
                          │  NAMENODE / MASTER                   │
                          │  RAM: namespace + file→blocks +      │
                          │       block→locations                │
                          │  (NO file bytes ever pass through)   │
                          └───────▲───────────────┬──────────────┘
       (1) getBlockLocations              (2) block handle +
           ("/data/big.log",                  replica list, sorted
            offset=70,000,000,                by NETWORK DISTANCE
            length=4096)                      from the client
           ~200 bytes, ~1 ms                  ~500 bytes
                  │                               │
                  │                               ▼
            ┌─────┴──────────────────────────────────────────┐
            │  CLIENT (a library inside the application)     │
            │  block index = floor(70,000,000 / 134,217,728) │
            │              = block 0                          │
            │  offset inside block = 70,000,000               │
            │  caches this location map for ~ minutes         │
            └─────┬──────────────────────────────────────────┘
                  │ (3) readBlock(handle, off=70,000,000, len=4096)
                  │     chosen replica = the CLOSEST one
                  ▼
   ┌───────────────────────┐   ┌──────────────────┐   ┌──────────────────┐
   │ DataNode 7            │   │ DataNode 12      │   │ DataNode 30      │
   │ rack A (SAME rack     │   │ rack B           │   │ rack B           │
   │ as client → chosen)   │   │ (fallback)       │   │ (fallback)       │
   │ block file on ext4    │   │                  │   │                  │
   │ + .meta CRC32C file   │   │                  │   │                  │
   └───────┬───────────────┘   └──────────────────┘   └──────────────────┘
           │ (4) verify CRC-32C for the covering 512-byte units,
           │     then stream 4096 bytes straight to the client socket
           ▼
        CLIENT  ── total ≈ 1 ms (metadata, often cached → 0) + ~10 ms (disk seek)
                              + ~0.1 ms (transfer) ≈ 11 ms cold, ~1 ms warm

Six things in that picture are load-bearing, and each is a design decision you should be able to defend:

4.4The Write Path (replication pipelining)#

Writing must update all R replicas consistently:

1. Client -> Master: "where do I write chunk C?" Master designates one replica as the PRIMARY
   (grants it a lease) and returns primary + secondaries.
2. Client PUSHES the data to all replicas in a PIPELINE (DN1 -> DN2 -> DN3), streaming
   node-to-node to use everyone's bandwidth efficiently (data flow decoupled from control flow).
3. Client -> Primary: "commit this write."
4. Primary assigns a serial ORDER to the mutation, applies it, and forwards the order to secondaries.
5. Secondaries apply in the same order and ACK the primary. Primary ACKs the client.
   (The lease + primary-assigned order = a mini single-leader consensus per chunk, File 08.)

The primary-per-chunk lease gives a consistent write order without global coordination; data is pipelined for throughput while control (ordering) is centralized at the primary.

Here is the same write drawn out, with the two flows — data and control — deliberately separated, because that separation is the clever part:

   CLIENT
     │  (1) "where do I write chunk C?"                 ┌──────────────┐
     ├─────────────────────────────────────────────────►│   MASTER     │
     │  (2) primary = DN1 (lease, 60s) ; secondaries    │  grants LEASE│
     │◄─────────────────────────────────────────────────┤  (§2.8)      │
     │      = DN2, DN3 ; client caches this             └──────────────┘
     │
     │  === DATA FLOW (bulk bytes, a linear PIPELINE, no master) ===
     │
     │  (3) push 64 MB to the NEAREST replica only
     ├────────────►┌─────┐  forwards while still receiving  ┌─────┐        ┌─────┐
                   │ DN1 ├─────────────────────────────────►│ DN2 ├───────►│ DN3 │
                   └─────┘                                  └─────┘        └─────┘
                   data sits in each node's LRU BUFFER, not yet committed
     │
     │  === CONTROL FLOW (tiny messages, ordering) ===
     │
     │  (4) "commit" ─────────────────► DN1 (PRIMARY)
     │                                   assigns serial number 4,271 to this
     │                                   mutation; applies it to its own disk
     │                                   ├──► DN2: "apply mutation 4,271"
     │                                   └──► DN3: "apply mutation 4,271"
     │                                   DN2, DN3 apply IN THAT ORDER, ack
     │  (5) ack / error ◄────────────── DN1

Why push the data down a chain instead of the client sending to all three? Bandwidth arithmetic. If the client sent 64 MB to each of three replicas itself, its uplink carries 192 MB and the write takes 192 MB ÷ (client uplink). With a 1 Gbps (125 MB/s) client link that is 1.54 seconds. In the pipeline the client sends 64 MB once — 0.51 s — and DN1→DN2→DN3 forwarding happens over the datacentre fabric in parallel with the client still sending, because each node begins forwarding bytes the instant it receives them rather than waiting for the whole chunk (this is store-and-forward vs cut-through, and cut-through is why the pipeline's total time is roughly "one transfer plus a small per-hop latency", not "three transfers"). GFS's paper gives the model: transferring B bytes to R replicas over a network with per-hop latency L takes about B/T + R×L, where T is the link throughput — so with B = 64 MB, T = 100 MB/s, R = 3, L = 0.5 ms, that is 640 ms + 1.5 ms ≈ 642 ms, essentially the cost of one transfer.

Why choose the chain order by proximity rather than by role? The client pushes to the nearest replica — which need not be the primary — and each node forwards to the nearest node that has not yet received the data. Decoupling the data path from the control path means the byte-shipping topology can be optimized purely for network distance while the ordering authority stays wherever the lease happens to live. This is the same idea as separating "who decides" from "who carries" that you saw in §2.1.

What the serial number buys. Concurrent writers to the same chunk region are the hard case: two clients writing overlapping ranges must end up with all three replicas agreeing on which write won. The primary imposes a single total order by assigning consecutive serial numbers, and every secondary applies mutations in serial-number order. Without it, DN2 might apply A-then-B while DN3 applies B-then-A, and the replicas would diverge permanently with no way to tell which is right. This is single-leader replication (File 08) scoped to one 64 MB chunk for 60 seconds — the coordination cost is amortized over an entire chunk's worth of writes instead of paid per write, which is precisely why GFS did not need a full consensus protocol on the data path.

What breaks, and what GFS actually promises. If a secondary fails to apply the mutation, the primary reports an error to the client, and the client retries — which means the successful replicas may have applied the data more than once. GFS therefore does not promise that a failed-then-retried write leaves byte-identical replicas; it promises the weaker model of §4.5. This is the single most surprising thing about GFS to a reader coming from POSIX, and it is deliberate: the alternative (a real atomic commit across replicas per write) would have cost a consensus round per mutation.

4.5GFS's Consistency Model — "consistent", "defined", and the record-append compromise#

Most treatments skip this, and it is exactly where a strong candidate separates from a memorizer. GFS defines two properties for a region of a file after a mutation, and you need both words:

Those give four outcomes, and the table is the answer to "what does GFS guarantee?":

Why "consistent but undefined" happens. A large write spanning chunk boundaries is split by the client into multiple chunk-level operations. Another client's concurrent large write is split the same way. Each chunk orders its own mutations independently, so chunk 1 might order A-then-B while chunk 2 orders B-then-A. Every replica of every chunk agrees (consistent), but the file as a whole contains a shuffled mixture of A's and B's fragments (undefined). The practical rule GFS imposed on its users: do not do concurrent overlapping writes; use record append instead.

Sub-Concept: Record Append. record_append is GFS's atomic-append primitive, and it exists because GFS's dominant workload was "hundreds of producers concurrently appending to one log file" (crawl results, click logs) — the write pattern that File 07's Kafka would later make its entire product. In an ordinary POSIX world each producer would have to seek to the end and write, requiring a distributed lock on "the end of the file." GFS instead lets the client say "append this record somewhere at the end and tell me where it landed." The primary picks the offset — checking that the record fits in the current chunk (records are limited to one quarter of the chunk size, i.e. 16 MB with 64 MB chunks, to bound the waste in the next step); if it does not fit, the primary pads the current chunk to its end, tells all replicas to do the same, and instructs the client to retry on a new chunk. Then it assigns the offset and all replicas write there.

The guarantee is "at least once, at the same offset." If any replica fails, the client retries, and the retry writes at a new offset — so successful replicas may now contain a duplicate of the record, and failed replicas contain a partial or padded region. GFS says: the record is guaranteed to appear at least once, atomically, as a contiguous run, and there may also be padding and duplicate fragments in the file. The application must cope, and Google's did, using exactly two techniques you should name: checksums inside each record so a reader can skip padding and detect torn fragments, and unique record IDs so a reader can deduplicate — which is the identical idea to the idempotency keys of File 15 and Kafka's producer IDs in File 07. Pros: no distributed lock, no seek coordination, extremely high concurrent-producer throughput. Cons: the file may contain garbage and duplicates, so it is unusable by naive readers; a "file" becomes a stream of self-describing records rather than an exact byte sequence. When to use: exactly the multi-producer log case, which is common enough that an entire product category was later built for it.

HDFS took the opposite decision, and knowing the contrast is valuable: HDFS supports only single-writer, append-only files — one writer holds a lease on the file (§2.8), writes sequentially, and no one else may write until it closes or the lease expires. No concurrent appends, no record append, no random writes at all (early HDFS did not even support append; it was added in 0.21). The trade: HDFS gives up GFS's multi-producer throughput and in exchange gives readers a simple, sane, exactly-once byte stream with no padding or duplicates — a far better substrate for the MapReduce/Hive/Spark ecosystem that grew on top of it. Object stores take the most extreme position of all: no appends whatsoever, whole-object immutable PUTs only (§2.5, §5.4), which is the strongest simplification and the reason their consistency story (§8) is comparatively easy.

4.6The Master Bottleneck & Its Mitigations#

A single master is elegant (one source of truth, simple consistency) but is a SPOF and a scaling limit (all metadata in one machine's RAM; all metadata ops through it). Mitigations, which every serious system evolved:

Sub-Concept: Write-Ahead Log (WAL) / Operation Log. Before the master applies a metadata change, it appends the change to a durable, replicated log on disk first (the WAL), then applies it. If the master crashes, it replays the WAL (plus the latest checkpoint/snapshot) to reconstruct its exact in-memory state — nothing is lost. This "log-before-apply, replay-on-recovery" pattern is the universal durability mechanism across databases (File 08's replicated log), filesystems, and Kafka (File 07's log is this idea). WAL + periodic checkpoint = fast, durable recovery.

4.7NameNode High Availability, Step by Step — and the exact metadata scaling ceiling#

"HA NameNode" is a phrase people repeat without knowing the machinery. Here it is concretely, because it is a compact, real example of File 08/File 18 consensus applied to a storage system, and interviewers love it.

The starting problem. The original HDFS NameNode was a single point of failure: it held the namespace in RAM, persisted changes to a local **edits log (the WAL) and periodic fsimage checkpoints, and if the machine died the cluster was down until an operator restored the files onto a new box — typically 30+ minutes, sometimes hours, because replaying a large edits log and then re-collecting block reports** from every DataNode is slow. Note the subtlety that makes it slower than you'd guess: the block→location map is not persisted (§4.1), so a restarted NameNode must wait for every DataNode to report its blocks before it can safely serve reads, and until a configured fraction have reported it stays in safe mode (read-only, no replication decisions). hdfs dfsadmin -safemode get shows this state, and a NameNode stuck in safe mode after a restart is one of the most common HDFS operational incidents.

The 2.x solution, component by component.

The result: failover in tens of seconds instead of tens of minutes, with no operator involvement. The residual cost: you now operate 3–5 JournalNodes and a ZooKeeper ensemble; failovers can flap if health checks are tuned too aggressively; and clients need retry logic that follows the failover (HDFS clients are configured with both NameNode addresses and a ConfiguredFailoverProxyProvider that retries the other one on failure).

The ceiling that HA does not remove. HA gives you availability; it does nothing for capacity, because the Standby is a copy of the same single machine's RAM. The arithmetic is well known in the Hadoop community and worth memorizing: the NameNode needs roughly 150 bytes of heap per namespace object, where a file with one block counts as about two objects (the inode plus the block), and each replica adds bookkeeping. As a rule of thumb, 1 GB of NameNode heap supports roughly 1 million blocks. So:

The wall is the JVM heap and its garbage collector, not the RAM price — that is the precise, senior-level statement. The two escapes are: HDFS Federation, where multiple independent NameNodes each own a disjoint slice of the namespace (/user on one, /data on another) over a shared pool of DataNodes, with a client-side ViewFS/router mount table stitching them into one apparent tree (the cost: federation is partitioning by subtree, so it does not help a single hot subtree and it does not give you cross-namespace atomic rename); and the deeper fix, replacing in-memory metadata with a distributed database, which is what Google's Colossus (§16.2) did with Bigtable and what modern systems like JuiceFS (metadata in Redis/TiKV) and Ozone (a scale-out Ozone Manager over RocksDB) do today. Object stores were designed this way from birth — their metadata layer is a sharded distributed index from day one (§5.8), which is why they never had a NameNode-shaped ceiling.

4.8Self-Healing (the failure-is-normal machinery)#

4.9The Small-File Problem — the arithmetic, and the five real fixes#

Everything in §4.2 was an argument that big blocks are cheap. Run that argument backwards and you get the most famous pathology of distributed filesystems, and one of the most reliable interview questions in this whole area.

The setup. You have 100 TB of data. Compare storing it as 800,000 files of 128 MB against 200,000,000 files of 512 KB. Same bytes. Now count the metadata objects, at the ~150 bytes/object and ~1 GB per million blocks figures from §4.7 (a file with one block ≈ 2 objects: the inode and the block):

The same 100 TB costs 250× more metadata, and 60 GB of live JVM heap puts you into the multi-second garbage-collection pause regime of §4.7 — where the NameNode intermittently stops answering every client. Note the second column too: a block does not shrink below its data, but its metadata cost is fixed regardless of size. A 512 KB file occupies 512 KB of disk (HDFS does not pad blocks) but a full block's worth of NameNode metadata. The block size is a maximum, not an allocation — a point candidates frequently get backwards, and stating it correctly is a good signal.

The three distinct costs, so you can name them separately:

The five real fixes, each with its trade-off:

The interview formulation: "The small-file problem is that metadata cost is per-file while storage cost is per-byte, so 100 TB in 200 million small files costs 250× the metadata of 100 TB in 800,000 large ones and blows the NameNode's heap — the fixes are to compact into container files, compact at ingest, or move to a system whose metadata layer is sharded rather than RAM-resident."

5Object Storage Internals (S3-style)#

Object stores like S3 apply the same principles with an HTTP/flat-namespace face and even bigger scale.

5.1Buckets, Keys, and the Flat Namespace#

To make the flatness concrete: when the S3 console shows you a folder icon labelled 2024/ inside photos/, no such entity exists anywhere in S3. There is no folder object, no folder metadata, no folder creation timestamp, and nothing to delete when it becomes empty (an "empty folder" simply ceases to be rendered, because folders are computed from the keys that exist). What actually happened is that the console issued a LIST with prefix=photos/ and delimiter=/, and S3 replied with a list of common prefixes — the distinct strings between the prefix and the next /. The folder is a rendering of a string operation. The one exception you will meet in practice is that some tools create a zero-byte object whose key ends in / (e.g. photos/2024/) purely so an empty "folder" shows up in a UI; that object is real, it is billed, and it sometimes confuses code that assumes every key names a file.

Three consequences fall out of flatness, and they are the reason it was chosen:

5.2LIST — Prefix, Delimiter, Pagination and the Continuation Token#

LIST is the operation people underestimate, so take it apart carefully. The modern call is **ListObjectsV2**, an HTTP GET on the bucket with query parameters. Every parameter matters:

The failure mode and its mitigations. LIST is slow (paginated at 1,000), expensive (billed per request — at $0.005 per 1,000 LIST requests, fully enumerating a 100-million-object bucket costs 100,000 requests ≈ $0.50 per full scan, which sounds trivial until a job does it hourly across many prefixes), and it is the operation most likely to be eventually consistent (§8.4). The mitigations, in order of preference: (1) Do not list — maintain your own index. Keep object keys in a database (Postgres/DynamoDB) written at upload time; query that. This is the single most important piece of practical advice in this section, and it is what every mature application does. (2) Use S3 Inventory, a managed daily/weekly report delivered as a CSV/ORC/Parquet manifest of every object in the bucket — one file to read instead of 100,000 requests. (3) Parallelize by prefix: if your keys begin with a hex character, issue 16 concurrent listings with prefix=0 … prefix=f. (4) Design keys so the prefixes you will need to scan are naturally narrow (tenant/{id}/date/{yyyy-mm-dd}/…).

5.3The Core Verbs — PUT, GET, HEAD, DELETE, COPY#

The API is deliberately tiny. Each verb, what it does, its exact limits, and its gotcha:

**PUT Object** — upload the whole object in one request; the body is the object. Limits: a single PUT is capped at 5 GB (above that you must use multipart, §5.8), and the maximum object size overall is 5 TB. Semantics: a PUT to an existing key is a full overwrite creating a new object (or, with versioning on, a new version), never a modification. Gotchas: the request must be authenticated with a signature over the request (§10.4), and if you supply Content-MD5 or x-amz-checksum-* the server verifies it and rejects a mismatch — which is how you get end-to-end integrity (§7.3). There is **no PATCH, no partial write, and no truncate.**

**GET Object — retrieve the object's bytes. Semantics:** returns 200 with the full body, or 206 Partial Content when a Range header was supplied (§5.4). Gotchas: the storage class matters — a GET against an object in Glacier Flexible Retrieval returns 403 InvalidObjectState rather than data, because that object must first be restored (§9.4). Cost is charged per request and per byte egressed (§9.7).

**HEAD Object — identical to GET but returns only the headers, no body**: size (Content-Length), Last-Modified, ETag, Content-Type, storage class, version ID, and user metadata. Why it exists and why it matters: it is the cheap way to ask "does this object exist and what is its size/version?" without paying to transfer it, and it is the operation a FUSE mount fires constantly for stat(), which is why mounting a bucket produces surprising request bills. It is billed at the GET request rate but transfers no data.

**DELETE Object — remove the object. Semantics that surprise people: DELETE is idempotent and always succeeds** — deleting a key that never existed returns 204 No Content, not 404, because the API's contract is "after this call, that key is absent," which is already true (this is precisely File 15's idempotency principle, and it makes client retries safe). With versioning enabled, DELETE does not remove anything: it inserts a delete marker as the newest version, so the object appears gone to normal GETs while all prior versions remain (and remain billed). Actually removing data then requires DELETE with an explicit versionId. **DeleteObjects (the plural, "multi-object delete") removes up to 1,000 keys in one request**, which is both cheaper and dramatically faster than 1,000 individual calls — always reach for it when cleaning up.

**COPY Object** (PUT with an x-amz-copy-source header) — a server-side copy, so the bytes never travel to your client and back. Limits: up to 5 GB in a single copy (larger objects need UploadPartCopy, §5.8). It is the mechanism behind "rename" (copy + delete), behind changing an object's storage class or encryption in place, and behind cross-region replication. The gotcha: a copy onto itself is how you change metadata/storage class, and it requires MetadataDirective=REPLACE, otherwise S3 rejects it as a no-op.

What is conspicuously absent, and why. No APPEND (except in Azure's append blobs, §15.1, and Google's compose), because appends require ordering coordination on a mutable tail (§4.5). No MOVE/RENAME, per §5.1. No server-side search or filter beyond prefix, because that would demand a secondary index over arbitrary attributes — which is a database, and the design decision was to not be one. No transactions across keys (§8.5). Every absence is a coordination cost the designers refused to pay, and each is the reason a corresponding higher layer exists. (The partial exception worth naming: S3 Select / Athena can push simple SQL projection and filtering of CSV/JSON/Parquet objects into the storage layer, which reduces egress but does not change the storage model.)

5.4Byte-Range GETs, ETags and Conditional Requests#

Range GETs. Although you cannot write part of an object, you can freely read part of one: send Range: bytes=1048576-2097151 and you get exactly that megabyte back with status 206. This is far more important than it sounds. It is what makes video seeking work (the player fetches only the segment around the requested timestamp), what makes columnar analytics viable (a Parquet reader reads the file footer to find column offsets, then range-GETs only the columns and row groups the query needs — often 2% of the file), and what enables parallel download: split a 10 GB object into 16 ranges, fetch them concurrently on 16 connections, and you multiply throughput because a single TCP connection to a single front-end is bandwidth-limited well below what the storage fleet can deliver (§11.3).

Sub-Concept: The ETag. An ETag (entity tag) is an HTTP response header carrying an opaque identifier for a specific version of a resource's content — if the content changes, the ETag changes. In S3 the ETag is, for objects uploaded in a single PUT with SSE-S3 or no encryption, exactly the hex MD5 of the object's bytes — so you can verify an upload by comparing it to a locally computed MD5. For a multipart upload it is deliberately not that: it is the MD5 of the concatenated binary MD5s of each part, followed by -<number of parts>, e.g. d41d8cd98f00b204e9800998ecf8427e-16. This trips up nearly everyone at least once: the ETag of a multipart-uploaded object does not equal the MD5 of its contents and can only be reproduced if you know the exact part size that was used. It is also not an MD5 at all for SSE-KMS or SSE-C encrypted objects. The rule: treat the ETag as an opaque change-detection token, and if you need a real content hash use the explicit x-amz-checksum-sha256 family (§7.3).

Conditional requests turn those tags into a concurrency-control mechanism, and there are four headers to know individually:

5.5Immutability, Versioning, Metadata and Tags#

Immutability, stated precisely. An object, once written, is never modified. There is no API to change one byte. An "update" is a new PUT that replaces the mapping key → object; the old bytes become garbage to be collected (or a retained version). Why the designers insisted on this — this is the payoff paragraph of §2.5:

The price is real and you must state it: read-modify-write of a large object costs a full re-upload; appending to a log means writing many objects and compacting them later; and a workload of frequent small updates is simply wrong for this system.

Versioning. Enabling versioning on a bucket makes every PUT to an existing key create a new version with a unique versionId rather than replacing the old one; GET without a version returns the newest; GET with versionId=… returns any specific one; DELETE inserts a delete marker (§5.3). What it buys: protection against the two failure modes that no amount of durability addresses — accidental overwrite and accidental/malicious delete (§0.8). What it costs: you are billed for every version's storage forever, so a client that re-uploads a 1 GB file hourly accumulates 8.7 TB per year. The mandatory mitigation: a lifecycle rule with NoncurrentVersionExpiration (e.g. "delete non-current versions after 30 days") and ExpiredObjectDeleteMarker cleanup. Also note the state machine: versioning can be enabled and then only suspended, never turned back off — existing versions persist forever until explicitly deleted. Its close relatives are MFA Delete (requires a multi-factor token to delete a version) and Object Lock (§10.6).

Metadata and tags — two different things people conflate.

5.6How Objects Map to Physical Storage#

Internally, an object store still: chunks large objects, replicates or erasure-codes each piece across many nodes/racks/AZs, keeps a metadata index (key → locations, version, checksum, size, ACL — Access Control List, the per-object list of who may read/write it) in a scalable, strongly-consistent store (often a sharded database / distributed KV — Files 04/08), and serves bytes from storage nodes. Consistent hashing (File 06) or a directory service maps keys/chunks to nodes.

Drawn out, an object store is three tiers, and naming them separately is how you describe one at a whiteboard:

                      ┌──────────────────────────────────────────────┐
   HTTPS request ───► │  FRONT-END / REQUEST-ROUTER FLEET            │
   PUT /bucket/key    │  · terminates TLS (File 01)                  │
                      │  · authenticates the SigV4 signature (§10.4) │
                      │  · evaluates bucket policy + IAM (§10.1)     │
                      │  · rate-limits, meters for billing           │
                      │  STATELESS → scale by adding boxes           │
                      └───────────────┬──────────────────────────────┘
                                      │
              ┌───────────────────────┴────────────────────────┐
              ▼                                                ▼
 ┌──────────────────────────────┐            ┌───────────────────────────────────┐
 │  INDEX / KEYMAP LAYER        │            │  DATA / STORAGE PLANE             │
 │  a sharded, replicated,      │            │  thousands of storage nodes with  │
 │  strongly-consistent,        │  points to │  dumb append-only extents;        │
 │  SORTED KV store:            │ ─────────► │  holds erasure-coded fragments    │
 │    key → {version, size,     │            │  or replicas (§6); verifies       │
 │    checksum, storage class,  │            │  checksums (§7); scrubs in the    │
 │    ACL, fragment locations}  │            │  background                       │
 │  sorted → prefix LIST is a   │            │  knows NOTHING about keys, users, │
 │  range scan (§5.2)           │            │  buckets, or policy               │
 │  partitioned by key range    │            │  placement spread across ≥3 AZs   │
 │  (Files 04, 06)              │            │  (§2.7)                           │
 └──────────────────────────────┘            └───────────────────────────────────┘

Why three tiers and not two? The front end exists because authentication, policy evaluation and billing are per-request concerns that have nothing to do with where bytes live — putting them on the storage nodes would couple a security change to a fleet of disk servers. The split between index and data plane is the §2.1 control/data separation again, and it has the same payoff: the index deals in small records with strong consistency requirements and the data plane deals in large opaque byte ranges with no consistency requirements at all (immutable data cannot be inconsistent), so each can be built with the technology suited to it — a consensus-replicated sharded database for one, a dumb log-structured extent store for the other. Say this in an interview and you have described S3, Ceph, Swift and MinIO all at once.

5.7The Metadata Index Problem#

At S3 scale (trillions of objects), the key→location index is itself a massive distributed database — sharded (File 04) and replicated with strong consistency (File 08). This is the "metadata server" from Section 4, grown into a full distributed system. Listing, versioning, and lifecycle rules all run against this index. The index is the hard part of object storage; the byte storage is comparatively easy.

Put numbers on "massive." S3 passed 100 trillion objects and regularly serves over 150 million requests per second at peak. An index with 10¹⁴ entries at, say, 300 bytes of metadata each is 30 petabytes of index alone — the metadata for S3 is larger than most companies' entire data estate. That index must be sorted (so prefix LIST is a range scan, §5.2), strongly consistent (so a PUT is immediately visible, §8.3), replicated across AZs (so it survives datacentre loss), and partitioned by key range with automatic splitting as any range gets hot or large (Files 04 and 06). It is, in other words, a planet-scale distributed database whose workload is 100% point-lookups and short range scans — and it is the reason object storage is hard to build and easy to use. Two practical consequences you should be able to name: key design controls index-shard placement, so keys sharing a long common prefix land on the same shard (§11.2); and listing cost is a property of the index, not the data, which is why S3 Inventory (a pre-computed manifest) exists to let you avoid touching the index at all.

5.8Multipart Upload & Large Objects#

Uploading a 5 TB object in one HTTP request is fragile. Multipart upload splits it into parts uploaded in parallel (throughput) and independently retryable (a failed part re-uploads without restarting the whole thing); the store assembles them on completion. This is chunking (2.2) surfaced to the client API.

The three-call protocol, named exactly. Multipart is not a flag; it is a small state machine you drive:

The limits, each with the reason it exists and what it implies:

The part-size arithmetic you should be able to do on the spot. Because parts are capped at 10,000, minimum part size = object size ÷ 10,000. For a 1 TB object that is 100 MB minimum; for a 5 TB object it is 500 MB. Choose too small and you exceed 10,000 parts and the upload fails at part 10,001 — after possibly hours of transfer. Choose too large and each retry after a network blip re-sends a huge amount of data, and you lose parallelism. The practical guidance: 8–16 MB parts for objects up to a few GB (small enough that a retry is cheap, large enough to amortize per-request overhead), scaling to 100–500 MB for multi-terabyte objects. The AWS CLI and SDK transfer managers default to an 8 MB part size with automatic escalation, and expose multipart_chunksize and multipart_threshold (default 8 MB) if you need to tune them.

Why the whole design is right: it converts one 5 TB request whose failure costs you 5 TB of re-transfer into 10,000 independent 500 MB requests whose failure costs you 500 MB — and simultaneously lets you use N connections to N different front-end servers, multiplying throughput past what one TCP connection can achieve (§11.3). It also lets you start uploading before you know the total size (a live stream), and UploadPartCopy lets a part be a byte range of an existing object, which is how you concatenate or reassemble large objects entirely server-side.

The failure mode that costs real money, and the mitigation. If a client starts a multipart upload and then dies — a crashed process, a killed CI job, a mobile client that lost signal — the uploaded parts remain stored and remain billed indefinitely, and they are invisible to LIST, so nobody notices. Large organisations routinely discover terabytes of orphaned parts years later. The mitigation is mandatory and takes one line of configuration: a lifecycle rule with AbortIncompleteMultipartUpload: DaysAfterInitiation: 7. You audit for the problem with ListMultipartUploads. Knowing this unprompted marks you as someone who has actually operated an object store.

5.9Storage Tiers & Lifecycle#

Object stores offer tiers trading cost for access speed/latency: hot (frequent access, higher cost) → infrequent access → cold/archive (Glacier) (cheapest, retrieval takes minutes–hours). Lifecycle policies auto-migrate objects between tiers by age (e.g., "move logs to archive after 30 days, delete after 365"). Cold tiers lean heavily on erasure coding (Section 6) for cheap durability since low-latency access isn't required.

5.10Presigned URLs & CDN Fronting#

Clients get presigned URLs (time-limited, signed links) to upload/download directly to/from the store without proxying bytes through your app (control/data-plane separation again). And the object store is the classic CDN origin (File 05): S3 behind CloudFront, with fingerprinted immutable keys → cache-friendly, globally fast. Storage (durability) + CDN (latency) is the standard media stack.

5.11The Life of a PUT and a GET, End to End#

Tie the tiers together by tracing both directions with real steps. **A 12 MB PUT photos/2024/cat.jpg:**

 1. DNS resolves bucket.s3.us-east-1.amazonaws.com → anycast/regional VIP (File 01, File 05)
 2. TLS handshake with a FRONT-END node; client sends the request with an
    Authorization: AWS4-HMAC-SHA256 ... header (a signature over method, path,
    headers, payload hash and timestamp — §10.4)
 3. FRONT-END: verify signature → look up bucket policy + IAM + block-public-access
    → decide ALLOW/DENY (§10.1). Deny = 403, and no storage is touched.
 4. FRONT-END streams the body; computes the checksum the client declared
    (Content-MD5 / x-amz-checksum-crc32c) and REJECTS on mismatch (§7.3)
 5. DATA PLANE: split into fragments, erasure-code (e.g. RS(6,3) → 9 fragments, §6.5),
    place fragments across ≥3 AZs honouring failure domains (§2.7); each fragment
    is written with its own checksum; wait for a durable quorum of writes
 6. INDEX: a single strongly-consistent commit of
       key "photos/2024/cat.jpg" → {version v3, 12 MB, etag, class STANDARD,
                                    fragment locations, SSE key id}
    ← THIS COMMIT IS THE MOMENT THE OBJECT EXISTS. Before it: nothing. (§8.3)
 7. 200 OK with ETag and x-amz-version-id. Old version v2 remains if versioning is on.

**And a GET photos/2024/cat.jpg:**

 1-3. Same DNS / TLS / signature / policy path.
 4.   INDEX lookup: key → {version, size, storage class, fragment locations}.
      If the class is Glacier Flexible Retrieval → 403 InvalidObjectState (§9.4).
 5.   DATA PLANE: read the k fragments needed (§6.6); if one node is slow, read
      an extra fragment from another and take whichever finishes first — the
      "hedged read" trick that turns tail latency into an extra bit of bandwidth.
 6.   Verify checksums; reconstruct if a fragment is missing (a DEGRADED READ);
      report any checksum failure so the repair pipeline re-creates that fragment (§7.5).
 7.   Stream bytes to the client. Egress is metered per GB HERE (§9.7).

Three observations worth carrying into an interview. First, the index commit is the atomicity boundary — data is written before the index points at it, so a crash mid-PUT leaves orphan fragments (later garbage-collected) but never a visible corrupt object. Second, every request is authenticated and policy-evaluated, which is why per-request cost is not zero and why a million-tiny-object workload is expensive in requests even when cheap in bytes. Third, the client never learns where the bytes live — unlike GFS, where the client is told the chunkserver list and connects to it directly. Object stores put a stateless proxy tier in the data path and pay a little latency for it, buying a security boundary, protocol independence, and the freedom to relocate data without any client knowing.

6Replication vs. Erasure Coding (durability math)#

How you achieve durability is a major cost/performance trade — and a strong senior topic.

This is the algorithm section of this file, so every scheme below gets the full treatment of RULE 4: mechanism, worked example with real numbers, pros with reasons, cons with failure modes, complexity, and the decisive when-to-use — followed by a comparison table and a plain-English decision rule in §6.9.

6.1The Idea Underneath Everything: k-of-n#

Before any specific scheme, get the shape of the problem. You have some data. You will store it as n pieces on n independent machines. Some of those machines will die. You want a rule that says: **as long as at least k of the n pieces survive, the original data can be rebuilt exactly. That is a k-of-n scheme**, and every redundancy technique in existence is one:

The two universal formulas, which you should be able to write without hesitating:

   storage overhead     = n / k
   failures tolerated   = n - k
   storage efficiency   = k / n

And the tension they encode: for a fixed number of tolerated failures n − k, you get cheaper storage by making k larger. Tolerate 2 failures with k=1, n=3 and you pay 3×. Tolerate 2 failures with k=10, n=12 and you pay 1.2×. The same fault tolerance for 40% of the cost — that is the entire commercial case for erasure coding, and it is why every hyperscaler uses it for the bulk of its bytes. So why does anyone still replicate? Because k is not free: as you will see in §6.6, **k is also the number of machines you must read from to repair a single lost fragment**, and often the number you must read from to serve a degraded read. Large k buys cheap storage and sells expensive repair. Every scheme below is a different position on that trade.

Note carefully what "independent" is doing in "n independent machines": it is the failure-domain requirement of §2.7, and the arithmetic of n − k tolerated failures is a lie the moment two of your n pieces share a rack.

6.2Algorithm 1 — Replication (N full copies)#

Store R full copies (typically R=3) on independent nodes/racks/AZs.

The mechanism in full, since "store 3 copies" hides real machinery. A write is acknowledged only when enough replicas are durable — GFS/HDFS pipeline to all R and wait (§4.4); Dynamo-style stores use a quorum, acknowledging after W of N replicas confirm and reading from R replicas, with W + R > N guaranteeing that any read set intersects any write set and therefore sees the latest write (File 08). Placement is failure-domain-aware (§2.7). A background process continuously compares the desired replication factor against the observed one — from heartbeats and block reports — and enqueues re-replication for anything under-replicated, prioritized by how under-replicated it is (a chunk down to one copy is far more urgent than one down to two, because the durability formula of §0.8 has the number of remaining copies in the exponent).

Complexity and cost, stated precisely. Space: R×, so 200% overhead at R=3 — for 10 PB of data you buy 30 PB of disk. Write cost: R× the bytes on the network and R× the disk writes, though pipelining means only 1× leaves the client (§4.4). Read cost: 1× — you read one copy, no computation at all, and you may read the nearest copy, which is a latency win, or several in parallel and take the fastest, which is a tail-latency win. Repair cost: 1× — copy one surviving replica, no math, and repair reads are sequential. CPU: essentially zero.

Pros, with the reason each is true. Reads are as cheap as possible because any single replica is a complete answer — no reconstruction, no fan-out, no cross-node coordination on the read path. Repair is minimal — restoring a lost 128 MB chunk moves exactly 128 MB, so a repair storm costs the network the same as the data lost, not a multiple of it. Locality is exploitable — you can place a replica close to a consumer (HDFS puts one on the writing node so MapReduce tasks read locally at zero network cost, the "move computation to the data" principle). Degraded operation is free — losing a replica does not make reads slower or more expensive, it only lowers durability. Simplicity — the code is easy to write, easy to reason about, and easy to debug at 3 a.m.

Cons, with the failure mode. The 3× bill is enormous at scale — for an exabyte, the difference between 3× and 1.5× is 1.5 exabytes of disk, hundreds of millions of dollars. It scales badly to higher fault tolerance — surviving 4 simultaneous failures needs R=5, i.e. 500% overhead, which nobody buys; erasure coding gets there for 1.4×. Correlated failure is the real killer — three copies in one rack is one copy (§2.7). Write amplification — R× the bytes written consumes SSD endurance (§0.4) and network.

When to use replication rather than erasure coding — the decisive conditions. (1) Small objects, because erasure coding a 4 KB object into 9 fragments produces 9 sub-kilobyte fragments, and per-fragment metadata and per-I/O overhead then dominate the data itself. (2) Hot data, where extra read fan-out and reconstruction latency would show up in user-facing p99. (3) Data that is written or overwritten frequently, since EC stripes must be recomputed on modification. (4) When repair bandwidth is your scarce resource, e.g. a cluster on modest networking. (5) When you want locality for computation. Almost every real system therefore uses replication for the hot/small tier and EC for the cold/large tier — which is §6.9's decision rule and exactly what Facebook did with Haystack → f4 (§12.1–§12.2).

6.3Why "Any k Fragments" Works — linear algebra over a Galois field, from zero#

Everyone repeats "any k of the n fragments can rebuild the data." Almost nobody can say why, and an interviewer who asks is separating the memorizers from the engineers. The answer is one idea from school algebra plus one adjustment for computers.

The school-algebra idea. If you have k unknowns, you need k independent linear equations to solve for them. Two unknowns x and y need two equations: x + y = 10 and x − y = 4 give x = 7, y = 3. A third equation like 2x + y = 17 is redundant — it adds no information if you already have two — but it is a perfect substitute for either of them. Lose the first equation and {x − y = 4, 2x + y = 17} still solves to the same x = 7, y = 3.

That is erasure coding. Treat your k data fragments as k unknowns d₁ … d_k. Manufacture n equations of the form fⱼ = a_{j1}·d₁ + a_{j2}·d₂ + … + a_{jk}·d_k, choosing the coefficients a so that every possible subset of k equations is independent and therefore solvable. Store one equation's result per node. Now any k surviving results give you k independent equations in k unknowns — solve, recover all the data. The n − k extras are pure redundancy.

**How the coefficients are chosen so that every k-subset works. The standard construction uses a Vandermonde matrix** — row j is [1, xⱼ, xⱼ², xⱼ³, …, xⱼ^{k-1}] for distinct xⱼ. Any k × k submatrix of a Vandermonde matrix is provably invertible when the xⱼ are distinct, which is precisely the "every k-subset is solvable" property we needed. (Equivalently, and this is the classical view: k data values define a unique polynomial of degree k−1, you evaluate that polynomial at n distinct points to make n fragments, and any k points determine the polynomial uniquely — the same fact that says two points determine a line and three determine a parabola. Practical implementations often use a systematic Cauchy or modified-Vandermonde matrix instead, for reasons in the next paragraph.)

Sub-Concept: Systematic codes. A code is systematic if the first k fragments are the original data itself, unmodified, and only the remaining m are computed parity. This is what every production system uses, and the reason is decisive: when nothing has failed — the overwhelmingly common case — a read is just "concatenate the k data fragments," with zero decoding math. Reconstruction cost is paid only in the rare degraded case. A non-systematic code would require solving the linear system on every single read, which would be insane.

Now the computer adjustment: why a Galois field. The algebra above assumed ordinary arithmetic over real numbers. Computers cannot use that: bytes are 8 bits, and 200 + 100 = 300 does not fit in a byte, while division produces fractions that do not exist in bytes. Rounding or overflow would destroy the exactness the whole scheme depends on. What is needed is an arithmetic that is closed (byte op byte = byte), exact (no rounding), and has all four operations including division so the matrix can be inverted.

Sub-Concept: Galois Field GF(2⁸) — arithmetic on bytes that actually closes. A finite field (or Galois field) is a finite set with +, −, ×, ÷ obeying all the usual algebraic laws. GF(2⁸) has exactly 256 elements — every possible byte — which is why it is the universal choice for storage codes. Its operations: - Addition and subtraction are both bitwise XOR. 0x53 + 0xCA = 0x99. Addition is its own inverse (a + a = 0), which is why XOR parity (§0.7) is the k=n−1 special case of Reed–Solomon. This makes addition free — one CPU instruction, and vectorizable across 32 bytes at a time with AVX2. - Multiplication treats each byte as a polynomial with 0/1 coefficients (0x53 = x⁶+x⁴+x+1), multiplies the polynomials, then takes the remainder modulo a fixed irreducible polynomial — for storage, almost always x⁸+x⁴+x³+x²+1 (0x11D), the same one AES uses. The modulo is what keeps the result inside one byte. Implementations do not do this arithmetically at runtime; they use 256-byte log/antilog tables (a×b = antilog[log[a] + log[b]]) or, on modern CPUs, the PSHUFB / GFNI / AVX-512 vector instructions that perform GF(2⁸) multiplies on 64 bytes at once. - Division exists for every non-zero element, which is exactly the property ordinary byte arithmetic lacks and exactly what lets you invert the k × k matrix to decode.

**The 2⁸ = 256 element count also sets a hard limit:** the Vandermonde construction needs n distinct non-zero field elements, so **n ≤ 255 in GF(2⁸)** — you cannot build a code with more than 255 total fragments without moving to GF(2¹⁶). Real configurations use n between 6 and 20, so this never binds in practice, but it is the correct answer to "is there a limit on n?"

What breaks: get the irreducible polynomial or the coefficient matrix wrong and the code silently fails to be MDS — some k-subsets become unsolvable, and you discover it only during a real multi-failure event. This is why nobody hand-rolls erasure coding; you use Jerasure, ISA-L (Intel's, which is what MinIO and Ceph use for its AVX-512 acceleration), **Klaus Post's reedsolomon** Go library, or Backblaze's Java implementation.

Sub-Concept: MDS (Maximum Distance Separable). A code is MDS if it achieves the theoretical optimum: with n fragments and k data fragments, it tolerates **exactly n − k losses** — the most any scheme can possibly tolerate, since you obviously cannot recover k unknowns from fewer than k equations. Reed–Solomon is MDS. Why the term matters: the Local Reconstruction Codes of §6.7 deliberately give up MDS-ness — they use a little more space than the theoretical optimum for the same fault tolerance — in exchange for far cheaper repair. Knowing that LRC is "non-MDS on purpose" is the crisp way to describe the trade.

6.4Algorithm 2 — Reed–Solomon Erasure Coding#

Sub-Concept: Erasure Coding (Reed–Solomon). Erasure coding splits data into k data fragments and computes m parity fragments (total n = k + m), spread across n nodes, such that the original data can be **reconstructed from any k of the n fragments. You can lose any m fragments and still recover. Example: Reed-Solomon(10,4) → 10 data + 4 parity = 14 fragments; tolerates any 4 losses; storage overhead is only 1.4×** (40%) instead of 3× — yet it tolerates more failures than 3× replication (which tolerates 2). This is the math behind cheap, durable storage.

Complexity, with real throughput numbers. Encode: computing m parity fragments from k data fragments is O(k × m) Galois-field multiply-accumulate operations per byte of stripe. With a table-based implementation that is roughly 1–2 GB/s per core; with Intel ISA-L using AVX2/AVX-512 vector instructions it reaches 5–15 GB/s per core, at which point encoding is essentially free relative to disk and network. Decode (degraded path): invert a k × k matrix once (negligible, done once per stripe geometry and cached) and then multiply — same order of magnitude as encoding, plus the dominant cost of **reading k fragments over the network**. Space: n/k. Network on write: n/k × the data (you ship 9 fragments for 6 fragments' worth of data at RS(6,3)) — notably less than replication's 3×, so EC writes are cheaper on the network as well as on disk. Network on repair: k × the lost fragment — the killer, dissected in §6.6.

6.5Worked Comparison — RS(6,3) against 3× Replication, with the chunk layout drawn#

Take a 6 GB object and store it both ways. Here is the physical layout, which is the picture to draw on a whiteboard:

=== 3x REPLICATION (a 1-of-3 scheme) =====================================
   object 6 GB
        │
        ├──► Node A  (AZ-1, rack 3)   [ full 6 GB copy ]
        ├──► Node B  (AZ-2, rack 9)   [ full 6 GB copy ]
        └──► Node C  (AZ-3, rack 4)   [ full 6 GB copy ]

   raw stored = 18 GB   overhead 3.00x   survives ANY 2 node losses
   read  = fetch 6 GB from any ONE node, no math
   repair one lost node = copy 6 GB from one survivor   (repair traffic 6 GB)

=== REED-SOLOMON RS(6,3) (a 6-of-9 scheme) ===============================
   object 6 GB  ──► split into k=6 DATA fragments of 1 GB each
                    then compute m=3 PARITY fragments of 1 GB each

     D1   D2   D3   D4   D5   D6      P1   P2   P3
     1GB  1GB  1GB  1GB  1GB  1GB     1GB  1GB  1GB
      │    │    │    │    │    │       │    │    │
     N1   N2   N3   N4   N5   N6      N7   N8   N9      <- 9 INDEPENDENT nodes
    AZ1  AZ2  AZ3  AZ1  AZ2  AZ3     AZ1  AZ2  AZ3      <- spread across AZs (§2.7)

   raw stored = 9 GB    overhead 1.50x   survives ANY 3 fragment losses
   healthy read = concatenate D1..D6 (systematic! zero math)   -> 6 GB read
   degraded read (D3 lost) = read ANY 6 of the surviving 8, decode -> 6 GB read
   repair one lost fragment = read 6 GB (six fragments!) to rebuild 1 GB
                              (repair traffic 6 GB for 1 GB lost = 6x amplification)

Now compare them head to head on the four axes that matter:

Read that table twice. RS(6,3) **halves the storage bill and tolerates one more failure than 3× replication. If storage cost were the only axis, replication would be indefensible. It is defensible only because of two numbers in that table: 6 nodes touched per read instead of 1, and 6× repair amplification. For 10 PB of stored data the money difference is stark: at $0.02/GB-month of raw disk, 3× replication costs $600,000/month and RS(6,3) costs $300,000/month**. Half of that is why hyperscalers invested so heavily in erasure coding.

Common real-world configurations, so the numbers are not abstract: Azure Storage originally used RS(6,3) and then moved to LRC(12,2,2) at 1.33× (§6.7); Facebook's f4 uses RS(10,4) at 1.4× and additionally XORs across datacentres; Backblaze's Vaults use RS(17,3) at 1.18× spread over 20 drives in 20 different chassis; Ceph's default EC profile is k=2, m=2 (2× — deliberately conservative for small clusters) but production clusters commonly run k=4, m=2 (1.5×) or k=8, m=3 (1.375×); MinIO defaults to half data, half parity (2×, extremely conservative) and lets you select EC:2 through EC:8; HDFS's built-in erasure coding (since 3.0) offers RS(6,3) and RS(10,4) policies.

6.6The Repair Amplification Problem — the reason erasure coding is not used for everything#

This is the single most important cost in erasure coding, it is what all subsequent code designs attack, and it is the thing candidates most often miss.

The mechanism. When a replicated chunk is lost, repair reads one surviving replica and writes one new copy — the network moves exactly as many bytes as were lost. When one RS(6,3) fragment is lost, you cannot reconstruct it from any single survivor, because every fragment is an independent equation and you need k equations to solve for the unknowns. So repair must **read k = 6 fragments from 6 different nodes, decode, and write 1 fragment. To restore 1 GB you moved 6 GB.** That ratio — k — is the repair amplification of §2.11.

Why it matters at scale, with arithmetic. Consider a cluster losing one 20 TB drive:

And now recall the durability formula of §0.8: **MTTR appears with exponent R−1, so a repair window six times longer is a directly worse durability posture — some of the fault tolerance you gained by going from 2 tolerated failures to 3 is given back by the longer window in which you are exposed. This is the honest, senior-level statement about erasure coding: it does not simply dominate replication; it trades storage cost for repair traffic and repair time.**

Three further costs of the same root cause:

Mitigations that do not change the code: spread each drive's fragments across thousands of distinct stripes so repair reads come from hundreds of source disks in parallel (declustered repair, §0.7) — this fixes repair time even though it cannot fix repair traffic; throttle repair bandwidth so it cannot starve production (§17.2); and prioritize stripes closest to unrecoverable. The mitigation that does change the code is §6.7.

6.7Algorithm 3 — Local Reconstruction Codes (LRC), the fix for repair amplification#

The one-line intuition. Add a few local parity fragments that each cover only a subset of the data, so the overwhelmingly common case — exactly one fragment lost — can be repaired by reading only that subset instead of all k.

Why it exists. Microsoft measured Azure Storage and found what everyone finds: over 98% of failure events are a single fragment loss. Paying k-fragment repair traffic for the 98% case in order to survive the 2% case is a bad bargain, and LRC (Huang et al., USENIX ATC 2012) restructures the code around that statistic.

The mechanism, with Azure's actual production configuration LRC(12,2,2). Take k = 12 data fragments. Split them into 2 local groups of 6. Compute:

   group 1                          group 2
   D1 D2 D3 D4 D5 D6                D7 D8 D9 D10 D11 D12
    └──────┬───────┘                 └───────┬────────┘
           ▼                                 ▼
          L1  (local parity)                L2  (local parity)

   D1 ... D12  ───────────────►  G1, G2   (global parities over ALL 12)

   total fragments n = 12 + 2 + 2 = 16      overhead = 16/12 = 1.33x

Repair of a single lost fragment. Say D3 dies. It belongs to group 1, so D3 is recovered from D1, D2, D4, D5, D6, L1 — 6 reads instead of 12. That is the payoff: repair traffic halved for the 98% case.

The comparison that makes the point. Compare LRC(12,2,2) with the MDS code of the same overhead, RS(12,4) at 16/12 = 1.33×:

Read the trade honestly. For the same storage cost, LRC halves single-failure repair traffic, and it pays by not being MDS: RS(12,4) survives any 4 losses, while LRC(12,2,2) is guaranteed only for any 3 — there exist specific 4-fragment loss patterns (e.g. losing enough of one group plus both globals) it cannot decode, though most 4-failure patterns are still recoverable. Microsoft's judgement was that halving the repair cost on 98% of events was worth a slightly weaker worst case, and given the same paper reports LRC saving substantial cross-node bandwidth in production, they were right for their workload.

Complexity: encode is O(k) extra work for the local parities — negligible. Repair for the common case drops from k reads to k/g reads where g is the number of local groups. When to use: any large-scale EC deployment where repair bandwidth is a real constraint — which is all of them above a few petabytes. When not to: small clusters where you have too few failure domains to place 16 fragments independently, and archival tiers where repair is rare enough that MDS optimality is worth more than repair speed. Where you will meet it: Azure Storage (its origin), HDFS-3's EC framework supports LRC-style policies, Ceph offers an lrc plugin alongside its jerasure and isa plugins, and Google's Colossus uses closely related ideas.

6.8The Rest of the Taxonomy — the codes and hybrids you should be able to name#

To satisfy the "know the whole taxonomy" bar, here is the remainder of the family, each with its one decisive property:

6.9The Trade-off Summary#

Real systems mix: hot data on replication (fast, cheap recovery), cold data erasure-coded (cheap storage). Facebook's HDFS-RAID, Azure LRC, S3, Ceph, and Backblaze all use erasure coding for the bulk. Advanced codes (LRC — Local Reconstruction Codes) reduce reconstruction cost by adding local parities so a single loss reads fewer fragments.

6.10The Full Comparison Table and the Decision Rule#

The decision rule — say this out loud:

7Data Integrity — Checksums, Bit Rot, and Background Scrubbing#

Redundancy (§6) protects you against a disk that announces its failure. This section is about the other kind: the disk that lies. It is the failure mode people forget in interviews, and at petabyte scale it is a certainty, not a curiosity.

7.1Why "the disk returned data" is not the same as "the data is correct"#

A storage stack has many places where a bit can flip and no error be reported: the drive's own controller firmware can have a bug and write a correct block to the wrong location (a misdirected write); a write can be acknowledged and never actually reach the platter after a power loss (a lost write or phantom write); a bit can decay on the platter or leak out of a flash cell over years (bit rot proper); a cosmic-ray-induced flip can corrupt data sitting in a server's RAM before it is ever written (which is why ECC memory exists); a marginal cable, a failing HBA, or a switch with a bad buffer can corrupt bytes in flight past the reach of TCP's checksum. In every one of these cases the read returns status "OK" and hands you wrong bytes. That is why the phenomenon is called silent data corruption (SDC).

The drive's own ECC (error-correcting code) per sector catches and corrects most media-level errors, and when it cannot correct it usually reports a URE (§0.7) — a loud failure, which is fine, because loud failures trigger repair. Silent corruption is the residue that ECC does not catch.

7.2The arithmetic at petabyte scale — why this stops being theoretical#

Two published studies are the ones to cite. CERN ran a write-then-verify campaign across ~3,000 nodes in 2007 and found silent corruption in roughly 1 in 10⁷ blocks — 22 corrupted files out of 33,700 written in one test. A CMU/NetApp study of 1.5 million drives over 41 months found that 0.466% of enterprise drives and 0.86% of nearline drives developed at least one silent checksum mismatch per year, and — the important detail — that errors were strongly spatially and temporally clustered, meaning a drive that produces one is very likely to produce more, so "one bad sector" is a signal to distrust the whole drive.

Turn that into your own numbers. Suppose you store 10 PB and read it all once a year, and take a conservative undetected-error rate of 1 per 10¹⁵ bits read:

   10 PB = 10 x 10^15 bytes = 8 x 10^16 bits
   expected silent errors per full pass = 8 x 10^16 / 10^15 = 80

Eighty silently corrupted reads per year, none of which announce themselves. Now note the compounding disaster: without checksums, your storage system will happily replicate the corrupt block to restore replication factor, and your backup will faithfully copy the corruption into every retained generation, so by the time a human notices the file is broken, every copy and every backup is broken too. This is the single strongest argument for checksums: not that they let you fix corruption, but that they stop you from propagating it.

7.3Where checksums live, and the case for end-to-end#

§2.9 covered the algorithms (CRC-32C, MD5, SHA-256) and granularity. The design question here is who computes and who verifies, and the principle is the end-to-end argument (Saltzer, Reed and Clark, 1984): a correctness check placed at an intermediate layer cannot guarantee correctness of the whole path, so the check must be performed at the endpoints.

Concretely, if the server computes the checksum after receiving your upload, then a bit that flipped in your application's memory, in your NIC, or in a middlebox is checksummed as correct and stored perfectly forever. If the client computes the checksum over the original buffer and sends it with the request, the server can verify it against what it actually received and reject the upload — closing the whole path. That is why S3 accepts:

On the storage side, the store keeps its own checksum per fragment/block and verifies it on every read (§4.8) — HDFS with CRC-32C per 512 bytes, Ceph with per-object and per-chunk checksums, ZFS with a per-block checksum stored in the parent block pointer (a design detail worth knowing: storing the checksum away from the data it protects is what lets ZFS detect misdirected writes, since a block written to the wrong place will not match the checksum recorded by whoever points at that place).

When you download, close the loop: request the stored checksum (ChecksumMode=ENABLED on GET returns it) and verify locally. A full end-to-end story is client computes on upload → server verifies → server stores and re-verifies on every read and during scrubs → client verifies on download.

7.4Background Scrubbing — finding corruption before you need the data#

The problem scrubbing solves. Verify-on-read only finds corruption in data you read. Cold archival data may sit untouched for years, quietly rotting, and you would discover the loss at the worst possible moment: during a restore, or during a repair when this copy was supposed to be the good one. Worse, corruption accumulates — if replica A rots in year 1 and replica B rots in year 3, and you never checked, you may reach the point where no good copy remains.

The mechanism. A low-priority background process continuously walks all stored data, reads each block, recomputes its checksum, and compares. Concretely, in Ceph the OSD daemon performs a shallow scrub (daily by default — compares object metadata and sizes across replicas, cheap) and a deep scrub (weekly by default — reads and checksums the actual data, expensive), controlled by osd_scrub_min_interval, osd_deep_scrub_interval, osd_scrub_begin_hour/end_hour to confine it to off-peak windows, and osd_scrub_load_threshold so it backs off when the machine is busy. HDFS runs a **BlockScanner** on every DataNode, configured with dfs.datanode.scan.period.hours (default 504 hours = 3 weeks) and a bandwidth cap dfs.block.scanner.volume.bytes.per.second (default 1 MB/s per volume). ZFS exposes it as zpool scrub.

The scrub-rate arithmetic you should be able to do. How fast must you scrub? Fast enough that a given block is verified more often than the mean time between corruptions, and fast enough that you find the second corruption before it becomes the last copy. To scrub 20 TB per drive every 3 weeks: 20 × 10¹² ÷ (21 × 86,400) = 11 MB/s per drive — about 5% of the drive's sequential bandwidth, which is why scrubbing is affordable. Push it to weekly and it is 33 MB/s, over 15% — noticeable. The trade is explicit: faster scrubbing detects corruption sooner (better durability) at the cost of I/O bandwidth stolen from production.

What breaks. Scrub storms: an unthrottled deep scrub across a cluster produces a large, sustained read load that shows up as elevated p99 latency for users — the classic Ceph complaint, diagnosed with ceph pg dump | grep scrub and fixed with the interval/hour/threshold settings above. The opposite failure is scrubbing configured so conservatively that a full pass takes longer than the mean time to a second failure, which silently defeats the point.

7.5Repair on Detection — closing the loop#

Detection without repair is only slightly better than nothing, so the last step is automatic. When a checksum mismatch is found — by a read or by a scrub — the sequence is:

The tie-back and the interview line: "Checksums turn silent corruption into loud corruption; redundancy turns loud corruption into a repair; scrubbing makes sure the corruption is found while a good copy still exists. You need all three — any two of them leaves a hole."

8Consistency in Object Storage — What S3 Promises, and What It Still Does Not#

8.1The vocabulary, defined before it is used#

Consistency here means: given that a write happened, what are the guarantees about what subsequent reads return? Three levels matter for storage, and they must be distinguished precisely (File 08 and File 19 develop this fully; this section is the storage-specific application).

8.2The historical S3 behaviour, and why it was that way#

From 2006 until December 2020, S3's documented model was: read-after-write consistency for PUTs of new objects, and eventual consistency for overwrite PUTs, for DELETEs, and for LIST — with one notorious asterisk: if you issued a GET or HEAD on a key before it existed (getting a 404) and then created it, even the new-object guarantee was void, because the negative result had been cached, and a subsequent GET could keep returning 404 for a while.

Why the design was like that is worth understanding rather than dismissing, because it is the same trade you will make in your own systems. S3's index is a geographically distributed, multi-AZ, sharded database (§5.7). Making every write synchronously visible at every replica of every index shard requires a consensus round across AZs on the write path (File 18), which costs an inter-AZ round trip (~1–2 ms) plus the coordination protocol, and it means a write fails if a quorum is unreachable. Eventual consistency let S3 acknowledge writes at whichever replica was reachable and propagate asynchronously, which bought lower write latency and higher availability under partition — the classic CAP choice of AP over CP (File 19). At S3's launch scale, with the technology of 2006, that was a rational trade.

What it cost users, in concrete failure modes: an ETL pipeline that wrote files and then listed the directory to find them would silently process an incomplete set — no error, just missing data; a service that overwrote a config object and immediately re-read it could get the old one; Hadoop's commit-by-rename protocol became unsafe, which is why the whole ecosystem of S3Guard (a DynamoDB table used purely as a strongly-consistent index of what S3 should contain, layered on top by the S3A connector), EMRFS consistent view, and Netflix's S3mper existed. These products were pure workarounds for eventual consistency, and their obsolescence is the clearest measure of how significant the 2020 change was.

8.3December 2020 — strong consistency for all operations, at no cost#

In December 2020 AWS announced that S3 provides strong read-after-write consistency automatically for all objects, in all regions, for all requests — GET, PUT, LIST, HEAD, and delete — with no performance penalty, no price change, no API change, and no opt-in. Every subsequent read (including a LIST) reflects every completed write.

How, mechanically? AWS has not published a full design paper, but the engineering blog and conference talks describe the shape, and it is the shape you would predict from §5.6: strengthen the metadata/index layer, since with immutable data (§5.5) the only thing that can be inconsistent is the key→object mapping. The approach uses a strongly consistent witness/cache-coherence mechanism — a lightweight, highly available component tracking recent writes for each key range, which the read path consults so it can never serve a version older than the last committed one, backed by replication protocols that make the index update atomic before the write is acknowledged. The lesson to carry away is architectural, not S3-specific: immutability moves the consistency problem entirely into a small metadata plane, where it becomes tractable to solve strongly — which is §5.5's payoff realized in production.

Where you still cannot assume it: other providers made the same move on different timelines (Google Cloud Storage has been strongly consistent for object operations and listing since 2018; Azure Blob has always been strongly consistent within a region), but self-hosted and S3-compatible systems vary, and a system's consistency for cross-region replication (§9.6) is always asynchronous — a write to us-east-1 is not instantly visible in eu-west-1, and no provider claims otherwise.

8.4Why LIST is the hardest operation to make consistent#

Take this apart, because it is a genuinely good interview question.

A GET key is a point lookup: route to the one index shard that owns that key, read the committed value. Making it strongly consistent means making one shard linearizable — a well-understood, local problem solved with a consensus group per shard (File 18).

A LIST prefix is a range query over a namespace, and that is categorically harder for three reasons:

This is exactly why "eventually consistent LIST" was the last thing to fall, and why table formats (§12.3) sidestep listing altogether by maintaining an explicit manifest of which files constitute a table rather than asking the store to enumerate a prefix.

8.5What strong consistency does not give you#

This is the part candidates over-claim, so be precise. Strong read-after-write consistency is a guarantee per object. It gives you nothing at all about relationships between objects. Specifically:

Interview line: "S3 is strongly read-after-write consistent per object since December 2020, which killed S3Guard — but it has never had cross-object transactions or cross-region consistency, which is exactly the gap Iceberg and Delta Lake fill with an atomic manifest pointer."

9Storage Classes, Lifecycle, and the Honest Cost Model#

The single most useful practical skill in this area is being able to compute what a storage design actually costs — including the two line items that are invisible in the sticker price and routinely dominate the bill.

9.1What a "storage class" is and what varies between them#

A storage class (Azure and GCS say access tier) is a per-object attribute selecting a different point on the cost/latency/availability curve. Crucially it is per object, not per bucket — one bucket can hold objects in every class — and it is set at PUT time or changed later by a lifecycle rule or a self-COPY. Five things vary, and you must check all five, because comparing classes on price alone is how people build expensive systems:

9.2The S3 classes, individually (us-east-1 list prices, the numbers to memorize)#

Notice the range: Standard is 23× the price of Deep Archive. Now the individual explanations, because each class exists for a specific reason:

The minimum-duration trap, worked: store a 1 GB object in Deep Archive and delete it after 10 days, and you are billed for 180 days — the early-delete charge is not a penalty bolted on, it is simply the remaining minimum. A lifecycle policy that moves objects to Deep Archive after 30 days and then deletes them at 60 days is a policy that pays 180 days of Deep Archive for every object, and it is a real and common misconfiguration.

The transition-cost trap: lifecycle transitions are billed per object (about $0.05 per 1,000 objects transitioned into Glacier classes). Transitioning 10 million small objects to Glacier therefore costs $500 in transition requests — possibly more than the storage you were trying to save. The rule: tier by bytes, not by object count; aggregate small objects before archiving them.

9.3Lifecycle policies and Intelligent-Tiering#

A lifecycle configuration is a set of rules attached to a bucket, each with a filter (prefix, object tag, size range) and one or more actions: Transition (change storage class after N days), Expiration (delete after N days), NoncurrentVersionTransition / NoncurrentVersionExpiration (the same for old versions, §5.5), AbortIncompleteMultipartUpload (§5.8), and ExpiredObjectDeleteMarker. Rules evaluate asynchronously once a day, so "delete after 30 days" means "some time on day 31," and you are not billed for the evaluation, only for the transitions it performs.

A canonical log-retention policy reads: transition to Standard-IA at 30 days, to Glacier Flexible at 90, to Deep Archive at 365, expire at 2,555 days (7 years); abort incomplete multipart uploads after 7 days; expire non-current versions after 30 days.

S3 Intelligent-Tiering is the managed alternative for when you cannot predict access patterns. It monitors each object's access and moves it automatically: Frequent Access → Infrequent (after 30 days no access) → Archive Instant Access (90 days), and optionally the asynchronous Archive Access (90+ days) and Deep Archive Access (180+ days) tiers. Moves back to Frequent happen instantly on access. Its cost: a monitoring and automation charge of about $0.0025 per 1,000 objects per month, and no retrieval fees on the automatic tiers. The decision rule: monitoring costs $2.50 per million objects per month, so Intelligent-Tiering pays for itself when objects are large enough that the storage saving exceeds that — roughly, when your average object is above ~128 KB and you genuinely cannot predict access. For a bucket of a billion tiny objects, the monitoring fee alone is $2,500/month and you should use an explicit lifecycle rule instead.

9.4The line items people forget: requests and egress#

Storage is the number on the pricing page. Requests and egress are frequently the larger part of the bill, and this is the single most valuable practical fact in this section.

Request pricing (S3 Standard, us-east-1): PUT, COPY, POST, LIST = $0.005 per 1,000; GET, SELECT, and all others = $0.0004 per 1,000. Lifecycle transitions and Glacier restores are billed separately and more expensively.

Data-transfer (egress) pricing on AWS: data in is free; data out to the internet is $0.09/GB for the first 10 TB/month, tapering to about $0.05/GB at very high volume; cross-region transfer is around $0.02/GB; cross-AZ within a region is $0.01/GB in each direction; transfer to CloudFront and then out of CloudFront to the internet is cheaper (CloudFront egress starts around $0.085/GB and drops with committed volume), and S3 → CloudFront origin fetches are free. There is also a permanent free tier of 100 GB/month out to the internet.

Now compare the magnitudes, which is the point. Storing 1 GB in S3 Standard for a month: $0.023. Serving that same 1 GB to the internet once: $0.09 — 3.9× the monthly storage cost for a single read. Serve it ten times and egress is 39× the storage.

9.5A worked total-cost example — a photo-sharing service#

Specification: 500 TB stored, 200 million objects averaging 2.5 MB, growing by 20 TB/month (8 million uploads), and serving 2 PB/month of downloads, of which 90% is cached by a CDN so only 10% reaches the origin.

STORAGE
  500 TB x 1,024 GB x $0.023                      = $11,776 / month

REQUESTS
  PUT: 8,000,000 uploads / 1,000 x $0.005         =     $40
  GET: origin fetches. 2 PB x 10% = 200 TB;
       200 TB / 2.5 MB = 80,000,000 GETs
       80,000,000 / 1,000 x $0.0004               =     $32

DATA TRANSFER  (this is the line item that decides the architecture)
  S3 -> CloudFront origin fetches (200 TB)        =      $0   (free)
  CloudFront -> internet, 2 PB x ~$0.06/GB
       2,048,000 GB x $0.06                       = $122,880 / month

  [Counterfactual: serve all 2 PB DIRECTLY from S3 at $0.09/GB
       2,048,000 GB x $0.09                       = $184,320 / month]
                                                   -----------
TOTAL with CDN                                    ≈ $134,700 / month
   of which storage is 8.7% and egress is 91%

Three conclusions to state out loud. (1) Egress is 91% of the bill and storage is under 9% — so optimizing storage class here is nearly pointless while a 5% improvement in CDN hit rate is worth $6,000/month. (2) The CDN saves ~$61,000/month and is the single highest-leverage component, which is the economic argument for File 05 that most people never make. (3) If this service instead had a write-heavy, low-read profile — backups, say — the ratios invert completely and storage class becomes everything. Always compute the ratio before optimizing; the answer differs by workload, and reciting "use IA to save money" without knowing the read pattern is exactly the mistake.

9.6Replication and geography as cost#

Cross-Region Replication (CRR) and Same-Region Replication (SRR) are rules that asynchronously copy new objects to a destination bucket. You pay: the destination storage (a second full copy), the cross-region transfer (~$0.02/GB), and a replication request charge per object. Replication Time Control (RTC) adds an SLA (99.99% of objects replicated within 15 minutes) for an extra per-GB fee. What replication buys that erasure coding cannot: survival of a whole-region event, and lower read latency for a geographically split audience. What it does not buy: protection from a bad delete, unless the destination has versioning plus object lock — by default, deletes can propagate. The interview point: replication is a geographic redundancy layer that sits on top of, and does not replace, the local durability scheme of §6.

10Access Control, Encryption, and the Classic Breach#

Object storage is the most-breached component in cloud computing, for a structural reason: it is the only storage type that is directly reachable from the internet by design. Every mechanism below exists because of a specific way that goes wrong.

10.1The three (and a half) authorization mechanisms, and which wins#

An S3 request is authorized by evaluating several policy types together. Knowing which is which — and the evaluation order — is a common interview question and a common production mistake.

The evaluation rule, stated exactly: an explicit **Deny in any applicable policy always wins; otherwise access requires an Allow** from at least one applicable policy (and, across accounts, from both sides — the resource policy and the caller's IAM policy); by default everything is denied. Memorize: explicit deny > explicit allow > implicit deny.

Block Public Access sits above all of it as a bucket-level (and account-level) override with four independent switches: BlockPublicAcls (reject new public ACLs), IgnorePublicAcls (ignore existing ones), BlockPublicPolicy (reject bucket policies that grant public access), and RestrictPublicBuckets (only allow AWS-service and authorized-user access even if a policy is public). Since April 2023 all four are on by default for new buckets — a direct product response to the breach history in §10.7. Turn them off only with a written reason, e.g. a genuinely public static website bucket.

10.2Presigned URLs — the mechanism, in full#

The problem. A user must upload a 2 GB video, or download a private file. Two bad options: give the browser AWS credentials (catastrophic — credentials are long-lived and grant far more than one file), or proxy the bytes through your application server (works, but now your servers carry 2 GB of traffic per upload, you pay for the compute and the bandwidth twice, and your app must handle streaming, retries and timeouts).

The solution. A presigned URL is an ordinary HTTPS URL with the authorization embedded in the query string as a signature, generated by a party who already holds credentials. Anyone holding the URL can perform exactly the signed operation until it expires, without ever seeing a credential.

The mechanism, concretely. Your backend, holding an IAM role, computes an AWS Signature Version 4 (SigV4) signature. It builds a canonical request — the HTTP method (PUT), the canonical URI (/bucket/uploads/abc123.mp4), the canonical query string, the signed headers (at minimum host), and a payload hash (UNSIGNED-PAYLOAD for presigned uploads) — hashes it with SHA-256, joins it with the timestamp and a credential scope (20260722/us-east-1/s3/aws4_request) into a string to sign, and then runs a chain of HMAC-SHA256 operations: signing key = HMAC(HMAC(HMAC(HMAC("AWS4"+secret, date), region), "s3"), "aws4_request"), and finally signature = HMAC(signing key, string to sign). The result is appended as query parameters:

https://bucket.s3.amazonaws.com/uploads/abc123.mp4
   ?X-Amz-Algorithm=AWS4-HMAC-SHA256
   &X-Amz-Credential=AKIA.../20260722/us-east-1/s3/aws4_request
   &X-Amz-Date=20260722T101500Z
   &X-Amz-Expires=900                <- 15 minutes
   &X-Amz-SignedHeaders=host
   &X-Amz-Signature=8f3d2a...

S3 recomputes the same HMAC chain with its own copy of the secret key and compares. Why this is secure without sharing a credential: HMAC is a keyed hash — you cannot produce a valid signature without the secret, and you cannot alter anything covered by the signature (the method, the path, the expiry, the region, any signed headers) without invalidating it. The date-scoped derived signing key means a leaked signature is useless for a different day, region, or service.

The trade-offs, stated honestly. Benefit: bytes flow directly between client and store — your servers never touch them — while authorization stays entirely in your application's hands. Cost 1 — it is a bearer token. Anyone who obtains the URL can use it; it appears in browser history, in server logs, in Referer headers, and in any chat where someone pastes it. Cost 2 — you cannot revoke it short of rotating the signing credential or deleting the object; expiry is the only control, which is why expiries should be minutes, not days. Cost 3 — the maximum lifetime is bounded by the credential: a URL signed by a temporary IAM role's credentials dies when those credentials expire, which can be much sooner than X-Amz-Expires claims — a genuinely confusing production failure. Cost 4 — an unconstrained presigned PUT lets the holder upload anything of any size to that key. Mitigation: use a presigned POST policy instead, which is a signed JSON document that can constrain content length (content-length-range), content type, key prefix, and required headers — the correct choice for browser uploads.

10.3Encryption — the four modes, and which to pick#

Two axes: at rest (bytes on disk) and in transit (bytes on the wire — always TLS 1.2+, and you should enforce it with a bucket policy condition "aws:SecureTransport": "false" → Deny). At rest, four modes:

Decision rule: default to SSE-S3 (free, on by default); use SSE-KMS with Bucket Keys when you need auditability, key-based revocation, or a compliance requirement naming customer-managed keys; use SSE-C only when a policy forbids the provider holding keys but you still want server-side encryption; use client-side when the threat model includes the provider.

10.4Object Lock, WORM, and legal hold — defending against yourself#

Durability does not protect data from an authorized DELETE (§0.8). S3 Object Lock does, by implementing WORM (Write Once, Read Many) semantics — the property, long required for financial and medical records, that once written a record cannot be altered or deleted for a defined period. Object Lock requires versioning and must be enabled at bucket creation. It offers:

Why this matters more every year: ransomware. An attacker with your credentials can delete or encrypt every object you can reach; versioning helps only if the attacker cannot also delete versions. Compliance-mode Object Lock plus versioning plus replication to an account with separate credentials is the actual backup architecture, and "we have eleven nines of durability" is not a backup strategy — a point worth making explicitly in an interview.

10.5The classic public-bucket breach — anatomy and prevention#

The pattern has repeated for a decade: Verizon (14 million customer records, 2017), Accenture (four buckets with internal keys and decryption master keys), the US Department of Defense's CENTCOM scraped-content archive, Dow Jones, Time Warner/NICE Systems, FedEx, Capital One (2019 — a slightly different shape: an SSRF against a misconfigured WAF yielded the EC2 instance's IAM role credentials, which had over-broad S3 permissions, and 100 million credit applications were exfiltrated), and hundreds of smaller incidents.

The mechanism is almost always one of four:

Prevention, in the order that actually reduces risk: leave Block Public Access on at the account level, disable ACLs entirely with Object Ownership enforced, use VPC endpoint policies to require that requests originate inside your network, scope IAM policies to specific prefixes with conditions, enable CloudTrail data events for S3 so object-level reads are logged at all (they are not by default — this is why breaches go unnoticed for months), enable S3 Access Analyzer which continuously reports any bucket reachable from outside your account, and enable default encryption and short presigned expiries. The tell of a mature answer in an interview is naming Block Public Access and CloudTrail data events specifically, rather than saying "make sure the bucket isn't public."

11Performance Engineering — Request Rates, Parallelism, and Moving a Petabyte#

11.1The request-rate model and key-prefix partitioning#

S3's documented performance is **at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes**. Both halves of that sentence matter.

Why a prefix is the unit. The index (§5.7) is a sorted, range-partitioned structure: contiguous runs of keys live on one partition, which splits automatically when it becomes hot or large. So parallelism across the index is parallelism across key ranges. Ten prefixes that S3 has split into ten partitions give you 55,000 GET/s; one prefix gives you 5,500.

The historical advice and what changed. Before July 2018, S3's partitioning adapted slowly, and because keys are sorted, a sequential or timestamp-prefixed key scheme (2026-07-22T10:15:00-event.json) sent every write to the same end of the keyspace and therefore the same partition — the identical hot-spot pathology as a monotonically increasing shard key in File 04 and File 17. AWS's official guidance was therefore to prepend a random hash to keys (7f3a/2026-07-22-event.json) to scatter them across partitions. In July 2018 AWS increased the per-prefix limits substantially and improved automatic adaptive partitioning, and retracted the randomization advice. What is still true in 2026: the limits are per prefix, so if you need more than 5,500 GET/s you must spread across multiple prefixes; adaptive splitting is reactive, taking minutes to respond, so a workload that goes from zero to 50,000 req/s in one second on a fresh prefix will see 503 SlowDown responses while the partition splits. Mitigations: design keys so natural parallelism exists ({tenant}/{date}/… gives you a prefix per tenant), ramp traffic gradually, and always implement exponential backoff with jitter on 503 SlowDown — the SDKs do this by default and disabling it is a mistake.

11.2Parallelism: multipart, byte-range, and connection count#

A single TCP connection to a single S3 front-end typically delivers 50–100 MB/s — limited by TCP window, round-trip time, and the front end, not by the storage fleet, which has enormously more capacity. Therefore all high-throughput object I/O is parallel I/O:

Transfer Acceleration is a different lever: uploads are routed to the nearest CloudFront edge and travel to the bucket's region over AWS's private backbone rather than the public internet. It costs $0.04–0.08/GB extra, and it only helps when the client is far from the bucket region and the public path is genuinely poor — AWS provides a speed-comparison tool precisely because it is often not faster. For same-region traffic it is pure waste.

11.3Moving a petabyte — the arithmetic that justifies shipping disks#

The question "how do we get 1 PB into the cloud?" is a standard interview prompt, and the answer is a division:

   1 PB = 1,000 TB = 8 x 10^15 bits

   Over a 10 Gbps link at 100% utilization:
      8 x 10^15 / 10^10 = 800,000 seconds = 9.26 DAYS
   At a realistic 60% sustained utilization:
      ~15.4 days
   Over a 1 Gbps link at 60%:
      ~154 days  (five months)
   Over a 100 Gbps link at 60%:
      ~1.5 days

And you must also ask what that link is for. Saturating your 10 Gbps internet uplink for two weeks means your business does not have an internet connection for two weeks — the real constraint is usually "we can spare 2 Gbps," which turns 1 PB into 77 days.

The physical alternative. AWS Snowball Edge (~80 TB usable per device), Snowmobile (a 45-foot shipping container holding up to 100 PB, towed by a truck), Azure Data Box (up to ~1 PB per Data Box Heavy), and Google Transfer Appliance (up to 300 TB). For 1 PB: order ~13 Snowball Edge devices, copy locally over 10–40 GbE inside your own datacentre (hours to a couple of days, at no cost to your internet link), ship them, and AWS ingests them. End-to-end: about a week, dominated by shipping, and it does not touch your uplink at all. The famous framing is worth remembering: "never underestimate the bandwidth of a station wagon full of tapes" — a truck carrying 100 PB across the country in 24 hours is roughly 9.6 Tbps of effective bandwidth, orders of magnitude beyond any affordable link, at the cost of terrible latency.

The decision rule: compute transfer days = data ÷ usable bandwidth. Under ~1 week → transfer over the network (with checksummed, resumable, parallel tooling). Over ~1–2 weeks → ship appliances. And for ongoing data, use AWS DataSync or Storage Gateway for continuous incremental sync, or a Direct Connect dedicated line if this is a permanent hybrid pattern — and note that Direct Connect also reduces egress pricing (~$0.02/GB instead of $0.09/GB), so at high sustained volume it pays for itself on cost alone.

12Adjacent Architectures — Small-Object Stores and Data Lakes#

12.1Haystack — deleting per-file metadata for billions of small blobs#

Facebook's 2010 Haystack paper is the canonical answer to §4.9's small-file problem, and its central insight is one sentence: the bottleneck for serving billions of small photos is not disk bandwidth, it is per-file metadata and the seeks it causes (§0.5).

The pre-Haystack architecture used NFS-mounted NAS filers, where reading one photo cost at least three disk operations — one or more to translate the path to an inode (deep directory trees made it worse), one to read the inode, one to read the data. At ~100 IOPS per spindle (§0.4), that is ~33 photos/second/disk. Haystack's design:

The trade-offs: the design only works for immutable, whole-read blobs; it gives up all filesystem semantics; and the in-memory index must be rebuilt from the volume file after a crash (mitigated by a periodically-checkpointed index file). The generalizable lesson, which is what an interviewer wants: when per-object overhead dominates, aggregate objects into large containers and keep a compact external index — the same move as SequenceFiles in Hadoop (§4.9), as MinIO and Ceph packing small objects into larger extents, and as any log-structured storage engine.

12.2f4 — warm storage, erasure coded#

Haystack replicates each volume (originally 3.6× effective, counting cross-datacentre copies). Facebook then observed that photo access rate falls by two orders of magnitude within a year, so paying hot-tier redundancy for old content is enormous waste. f4 (OSDI 2014) is the warm tier:

Takeaway: the hot→warm, replicate→erasure-code migration of §6.9's decision rule, implemented at exabyte scale, with the sealing step as the enabling mechanism. Immutability is what makes erasure coding affordable, which is §5.5's argument proved in production.

12.3Data lakes and table formats — putting ACID on top of immutable objects#

The problem. Object storage is a wonderful place to keep petabytes of Parquet files, and a terrible place to keep a table. Concretely, on raw object storage you cannot: atomically add or replace a set of files (§8.5 — no multi-object transaction); know which files constitute the table without an expensive LIST (§5.2); update or delete individual rows (GDPR "delete this user" against immutable objects); read a consistent snapshot while a writer is appending; or evolve a schema safely. Classic Hive solved "which files" by listing directories and used rename to commit — both of which are broken on object storage (§5.1).

The solution — the table format. A table format is a specification for metadata files, stored alongside the data files in the same bucket, that describe exactly which data files make up the table at each point in time, plus their schema and statistics. Commit becomes a single atomic pointer swap on one small metadata object — which object storage can do, either via a conditional write (§5.4) or an external catalog. The three implementations:

The comparison and decision rule. All three give you ACID, time travel, and schema evolution over object storage. Choose Iceberg as the default for new work, for engine neutrality and the broadest vendor support. Choose Delta Lake if you are a Databricks/Spark shop and want the deepest integration. Choose Hudi if your dominant pattern is streaming upserts and incremental CDC consumption. The anti-recommendation: using none of them — a "data lake" of bare Parquet files in prefixes, committed by directory rename — which is the classic setup that quietly produces incomplete query results and cannot delete a user's rows.

Tie-back: table formats are the clearest illustration of this file's central theme. Object storage deliberately removed transactions, hierarchy, rename and mutability to achieve its scale (§0.6, §5.5), and every layer that needs those properties back must rebuild them above the storage as metadata — an atomic pointer swap over immutable files. That is the same pattern as GFS's master over dumb chunkservers (§4.1) and as the index-over-data-plane split of §5.6.

7Pros, Cons & Trade-offs#

7.1Benefits#

7.2Costs & Trade-offs#

8Interview Diagnostic Framework (Symptom → Solution)#

15Real Implementations & Product Landscape#

Everything so far has been mechanism. This is the catalogue: what you can actually buy or install, what each thing is, what its distinguishing architectural choice is, what it is genuinely best at, what it costs, and — the part that matters — which one to pick in which situation. Every product named anywhere else in this file is defined here. The families run from managed object storage, through self-hosted object storage, distributed filesystems, managed file storage, and block storage for contrast, to the table formats that sit on top.

15.1Family 1 — Managed cloud object stores#

Amazon S3. Launched March 2006 as AWS's first public service, S3 is the product that defined the category, and its API is the de facto industry standard — so much so that "S3-compatible" is a product category. Architecturally it is the three-tier design of §5.6: a stateless authenticating front-end fleet, a sharded strongly-consistent index (§5.7, §8.3), and an erasure-coded data plane spread across at least three Availability Zones. Distinguishing mechanism: the sheer breadth of the surrounding platform — eight storage classes (§9.2), lifecycle, versioning, Object Lock, replication, Intelligent-Tiering, Access Points, S3 Select, Inventory, Event Notifications (to Lambda/SQS/SNS/EventBridge, which is what makes S3 an event source rather than just a store), and Mountpoint/S3 File Gateway for filesystem-ish access. Best at: being the default — the deepest ecosystem integration, the most mature feature set, and eleven nines. Limits: egress at $0.09/GB is the defining cost and the strategic lock-in (§15.2); per-request charges make tiny-object workloads expensive; and the feature surface is large enough that misconfiguration is a real risk (§10.5). Scale figures: 100+ trillion objects, 150M+ requests/second peak, 5 TB max object, 3,500 PUT/5,500 GET per second per prefix.

Google Cloud Storage (GCS). Google's object store, notable for three design differences from S3. First, strong consistency including list-after-write has been the guarantee since before S3 had it (2018), a direct benefit of building on Google's Spanner/Colossus lineage. Second, buckets are location-flexible in a cleaner way: a bucket can be regional, dual-region (two named regions with synchronous-ish replication and a single namespace), or multi-region (a continent), which gives cross-region redundancy without you configuring replication rules. Third, the storage classes (Standard / Nearline (30-day min) / Coldline (90-day) / Archive (365-day)) apply uniformly and switching class does not change the API or latency — even Archive reads in milliseconds, which is a genuine differentiator against Glacier's restore protocol (§9.2). It also has Autoclass (Intelligent-Tiering's equivalent) and an XML API that is largely S3-compatible, so migration is mostly a config change. Best at: analytics-adjacent workloads (BigQuery reads GCS natively), and multi-region simplicity. Limits: smaller ecosystem than S3; egress pricing is similar (~$0.12/GB to the internet, cheaper within Google's network).

Azure Blob Storage. Microsoft's object store, and the one with a taxonomy you must know because it is genuinely different: three blob types, chosen at creation and immutable thereafter.

Access tiers are Hot / Cool (30-day min) / Cold (90-day) / Archive (180-day, hours to rehydrate), and redundancy is chosen explicitly as LRS (3 copies in one datacentre), ZRS (3 availability zones), GRS (asynchronously to a paired region), or GZRS — a clearer model than most. Best at: Microsoft-ecosystem integration, hierarchical namespace via ADLS Gen2 (a flag that adds real directories with atomic rename to blob storage — which solves the §5.1 rename problem for analytics and is a genuine advantage for Hadoop/Spark workloads). Limits: the blob-type distinction is a trap for newcomers; API differs from S3 so tooling compatibility is weaker.

Cloudflare R2 — and why zero egress is strategically significant. R2 is an S3-API-compatible object store launched in 2022 whose headline is $0.015/GB-month storage and $0 egress — no charge to read data out, ever. Understand why that is a strategic weapon and not just a discount. Cloud egress fees are the primary mechanism of data gravity: once a petabyte lives in S3, moving it costs 1,000,000 GB × $0.09 = $90,000 in egress alone, and every architecture that reads it from outside AWS pays a recurring tax. That tax is what keeps compute co-located with storage and keeps customers inside one provider. R2 removes it, which (a) makes multi-cloud architectures economically viable, (b) makes object storage a first-class origin for content served by anyone's CDN, and (c) forced a visible industry response — AWS, Google and Azure all now waive egress fees for customers leaving the platform entirely, a change regulators pushed for and R2's pricing made commercially unavoidable. R2 charges instead for operations (Class A writes ~$4.50/million, Class B reads ~$0.36/million) and has a genuinely useful free tier. Best at: anything read-heavy and internet-facing — media, downloads, model weights, public datasets — where §9.5 showed egress is 91% of the bill. Limits: fewer storage classes (an Infrequent Access tier exists; no deep-archive equivalent), a smaller feature surface than S3 (no Object Lock parity historically, more limited lifecycle), a single global namespace with automatic location management rather than fine-grained region control, and you are adopting Cloudflare as a dependency. The decision rule is arithmetic: if monthly egress GB exceeds roughly a quarter of stored GB, R2's economics beat S3's decisively.

Backblaze B2. A storage-only company (famous for publishing its drive-failure statistics, which is where §0.8's AFR figures come from) offering an S3-compatible API at $6/TB/month ($0.006/GB) — about a quarter of S3 Standard — with free egress up to 3× your stored data per month and free egress entirely to partner CDNs (Cloudflare, Fastly, bunny.net) through the Bandwidth Alliance. Internally it uses Reed–Solomon(17,3) across 20 drives in 20 separate chassis in a "Vault" (§6.5). Best at: backup, archive, and media origin at a fraction of hyperscaler cost. Limits: far fewer features (no equivalent of the full lifecycle/analytics/event ecosystem), fewer regions, and no adjacent compute — so you will be paying to move data to wherever your compute is unless you use a partner CDN.

Wasabi. Positioned as "hot cloud storage": ~$6.99/TB/month, no egress fees, no request fees — deliberately the simplest possible pricing. The catch you must read: the free egress is bounded by a fair-use policy (monthly egress should not exceed your stored volume) and there is a 90-day minimum storage duration on every object, so short-lived data is billed for 90 days regardless. Best at: predictable-cost backup/archive where egress is moderate. Limits: no storage classes at all (one tier by design), fewer regions, and the fair-use policy makes it wrong for CDN-origin workloads with high egress ratios.

15.2Family 2 — Self-hosted object storage#

MinIO. A single Go binary that speaks the S3 API, released in 2015 and now the default way to run object storage yourself or inside Kubernetes. Distinguishing mechanism: radical operational simplicity plus per-object inline erasure coding. You start a server with a list of drives and MinIO forms erasure sets (groups of drives, sized 4–16), striping each object across a set with Reed–Solomon computed via Intel ISA-L (§6.4). Its default is half data, half parity (EC:N/2, hence 2× overhead, tolerating the loss of half the drives) with EC:2 through EC:8 selectable. Critically, erasure coding is per object, and parity is computed inline on write with no separate metadata database — object metadata lives beside the object data (in xl.meta), which is why there is no NameNode-shaped component to operate. It supports versioning, object lock/WORM, bucket replication, lifecycle, KMS integration, and bucket notifications. Best at: S3 compatibility on your own hardware or in Kubernetes; also the standard way to run S3 locally for tests. Limits: the storage layout is fixed at deployment — you scale by adding whole server pools, not individual drives, and rebalancing across pools is a deliberate operation; historically some enterprise features moved behind a commercial licence, and the AGPLv3 licence has real implications if you embed it. Reach for it when you want S3 semantics on-prem with minimal operational surface. Not when you need block and file and object from one system — that is Ceph.

Ceph — and CRUSH, the idea worth the section. Ceph (Sage Weil's 2006 doctoral work, now maintained by Red Hat/IBM) is the general-purpose distributed storage system: one cluster provides object storage (RADOS Gateway, S3- and Swift-compatible), block storage (RBD, which is what OpenStack VMs boot from), and a POSIX filesystem (CephFS) over the same underlying object store, RADOS. Components: OSDs (one daemon per disk, storing objects and handling replication, recovery and scrubbing), MONs (a small Paxos-replicated quorum holding the cluster map — File 18), MGRs (metrics and management), and MDSs (metadata servers, only for CephFS).

Sub-Concept: CRUSH — placement by computation instead of by lookup. Every system in this file so far answers "where does this data live?" with a lookup: GFS asks the master (§4.1), S3 asks the index (§5.7). That lookup is a component you must scale, replicate, and keep consistent. CRUSH (Controlled Replication Under Scalable Hashing) removes it. Instead of a directory, every client and every OSD holds a small cluster map — the failure-domain tree of §2.7 (root → datacenter → rack → host → osd, with a weight per device reflecting its capacity) — and a set of placement rules. To find an object's locations you: (1) hash the object name to a placement group (PG) id, pg = hash(name) mod pg_num — PGs exist so the system tracks millions of objects as thousands of groups, because per-object placement state would be unmanageable; (2) run the deterministic CRUSH function CRUSH(pg_id, cluster_map, rule) → [osd.17, osd.94, osd.203], which pseudo-randomly descends the tree, choosing at each level according to device weights and honouring the rule's failure-domain constraint ("pick 3 OSDs in 3 different racks"). Every participant computes the same answer from the same map with no communication at all.

What this buys: no metadata server on the data path, no lookup latency, and no central bottleneck — a client with the map can address any of billions of objects instantly. The property that makes it work under change is that CRUSH uses stable hashing internally (the straw2 bucket algorithm): adding or removing a device moves only the fraction of data proportional to the change, not everything — the same property as consistent hashing (File 06), extended to weighted, hierarchical, multi-replica placement. What breaks: the map must be distributed to everyone, so map updates (topology changes, OSDs going in/out) trigger peering and backfill, and a large topology change causes a substantial rebalance — this is the source of most Ceph performance incidents, mitigated with osd_max_backfills, norebalance flags during maintenance, and the upmap mechanism that pins specific PGs to override CRUSH's choice for fine-grained balancing. Also, **pg_num matters**: too few PGs gives uneven distribution, too many gives excessive per-PG overhead; the modern pg_autoscaler handles it. Tie-back: CRUSH is the clearest alternative in production to the metadata-lookup architecture of §2.1 — and knowing that this is the axis on which Ceph differs from GFS/S3 is exactly the kind of comparison senior interviews reward.

Ceph, continued. Best at: being the one storage system for a private cloud — the only mainstream open-source option delivering block, file and object from one cluster, with tunable replication or erasure coding per pool. Limits: it is operationally heavy — the learning curve is genuinely steep, tuning matters (PG counts, scrub windows §7.4, backfill throttles), small clusters underperform, and recovery events can hurt client latency. Reach for it when you are building private-cloud infrastructure and need all three paradigms. Not when you only need S3 semantics — MinIO or SeaweedFS will be a tenth of the operational cost.

SeaweedFS. An open-source system explicitly modelled on the Haystack paper (§12.1): a master manages volumes rather than files, volume servers hold large volume files packed with needles, and a filer layer adds an optional POSIX-ish namespace and an S3 API on top. Distinguishing mechanism: its metadata cost is per volume, not per file, so it handles billions of small files with modest memory — the direct architectural answer to §4.9. It supports erasure coding for warm data, tiering to cloud storage, and FUSE mounts. Best at: enormous numbers of small objects (thumbnails, tiles, documents) on your own hardware. Limits: a smaller community than Ceph or MinIO, S3 compatibility is good but not exhaustive, and the filer's metadata store is a separate component you must choose and operate (Redis, MySQL, LevelDB, TiKV…).

OpenStack Swift. One of the two original OpenStack projects (2010), and the oldest open-source object store. Distinguishing mechanism: a ring — a static, pre-computed partition-to-device mapping built by an administrator with an explicit replica count and failure-domain zones, rather than Ceph's computed CRUSH or a queried index. Swift was designed eventually consistent with replicas reconciled by background auditors and replicators. Best at: existing OpenStack deployments; it is stable and well-understood. Limits: its native API is not S3 (an S3 compatibility middleware exists), eventual consistency is a poor fit for modern expectations (§8.3), and momentum in the ecosystem has clearly moved to Ceph and MinIO. Do not choose it for greenfield work unless you are already committed to OpenStack.

Garage. A newer (2021, Deuxfleurs) lightweight S3-compatible store written in Rust, designed explicitly for geographically distributed, heterogeneous, low-power, unreliable nodes — self-hosting across several homes or small sites rather than a datacentre rack. Distinguishing mechanism: it assumes high and variable inter-node latency, avoids requiring a consensus quorum for the data path, and enforces zone-aware replication over nodes of very different capacities. Best at: small-scale, multi-site, self-hosted deployments where Ceph would be absurd. Limits: small feature set, small ecosystem, not intended for datacentre-scale performance.

15.3Family 3 — Distributed filesystems#

HDFS (Hadoop Distributed File System). The open-source GFS (§4), and still the substrate under a large amount of on-prem analytics. Architecture per §4.1: NameNode + DataNodes, 128 MB default blocks, 3× replication, rack awareness, HA via QJM + ZooKeeper (§4.7), and since Hadoop 3.0 native erasure coding policies (RS(6,3), RS(10,4)) that cut storage overhead from 3× to 1.5×/1.4× for cold data. Best at: high-throughput sequential scans over huge files co-located with compute (the "move computation to the data" model that made MapReduce work). Limits: the NameNode heap ceiling and small-file problem (§4.9), operational weight (ZooKeeper, JournalNodes, the whole Hadoop stack), and a fundamental economic problem — it couples storage to compute, so you scale both together, which is exactly what cloud object storage decoupled. The honest 2026 assessment: for new work, object storage plus a table format (§12.3) has displaced HDFS almost everywhere except existing on-prem estates.

CephFS is Ceph's POSIX filesystem, adding MDS (metadata server) daemons that hold the directory tree in memory and can dynamically subtree-partition the namespace across multiple active MDSs — a scale-out answer to the NameNode ceiling. Best at: POSIX shared storage in a cluster you already run Ceph on. Limits: MDS tuning is subtle, and metadata-heavy workloads still require care.

GlusterFS. A scale-out NAS that aggregates ordinary directories on ordinary servers ("bricks") into one volume. Distinguishing mechanism: it has no metadata server at all — file locations are computed by hashing the filename into a distributed hash ring (elastic hashing), and files are stored as whole files on the underlying local filesystems, so you can read your data with plain Unix tools if Gluster is unavailable — a genuinely valuable recovery property. Best at: simple scale-out NFS/SMB replacement for large files. Limits: small-file and metadata-heavy performance is poor (an ls may fan out to every brick), and Red Hat's investment has clearly shifted to Ceph; treat it as a mature but declining option.

Lustre. The dominant HPC parallel filesystem, running on a large fraction of the TOP500 supercomputers. Architecture: MDS/MDT (metadata servers and targets), OSS/OST (object storage servers and targets holding striped file data), and clients that, after one metadata call, read and write directly and in parallel to many OSTs over RDMA-capable networks (InfiniBand). Best at: raw aggregate bandwidth — terabytes per second across thousands of clients doing large coordinated I/O, which is exactly the HPC checkpoint/restart pattern. Limits: it is a demanding system to operate, historically fragile to node failure (no built-in replication — durability is delegated to hardware RAID on the targets), poor at small-file and metadata-heavy workloads, and it needs a specialized network. Available managed as AWS FSx for Lustre, which is how most people should consume it.

BeeGFS. Lustre's more approachable competitor (originally Fraunhofer): similar architecture (separate metadata and storage services, client-side striping), but explicitly designed for ease of installation and flexible deployment — services are user-space daemons, metadata can be distributed across many servers by directory, and you can add servers without redesigning the cluster. Best at: HPC and AI-training storage for teams without a dedicated Lustre administrator. Limits: smaller ecosystem; the enterprise features (high availability, quotas) are in the paid edition.

15.4Family 4 — Managed file storage#

AWS EFS (Elastic File System) is managed NFSv4.1, elastic (no provisioning — it grows and shrinks automatically), multi-AZ by default, and mountable by thousands of EC2/Lambda/ECS clients at once. Two throughput models matter: Elastic (pay per GB transferred, right for spiky workloads) and Provisioned (buy MB/s, right for steady ones), plus a legacy Bursting model where throughput scaled with stored size and exhausting burst credits was the classic EFS incident. Storage classes: Standard (~$0.30/GB-month) and Infrequent Access (~$0.016/GB-month plus a per-GB access charge), with lifecycle management between them. Best at: shared POSIX storage for containers and Lambda where you cannot use object storage. Limits: an order of magnitude more expensive than S3 (§3.6), per-operation latency in the low milliseconds (much slower than local disk), and it is still NFS — metadata-heavy workloads suffer.

AWS FSx is a family, and knowing that it is a family is the point: FSx for Lustre (HPC/ML, and it can transparently present an S3 bucket as a filesystem, importing and exporting objects — the standard pattern for training on S3-resident data), FSx for Windows File Server (real SMB with Active Directory integration and Windows ACLs), FSx for NetApp ONTAP (enterprise NAS features: snapshots, dedup, SnapMirror, multi-protocol NFS+SMB), and FSx for OpenZFS. Reach for FSx rather than EFS when you need SMB/Windows semantics, NetApp features, or Lustre-class bandwidth.

Google Filestore and Azure Files are the equivalents (managed NFS and managed SMB/NFS respectively); Azure Files additionally offers Azure File Sync, which turns Windows servers into caches of a cloud file share.

15.5Family 5 — Block storage, for contrast#

AWS EBS (Elastic Block Store). Network-attached block volumes for EC2, replicated within a single AZ (§3.5). The volume types are the thing to know:

Local NVMe instance storage (EC2 i3/i4i/im4gn/d3 families, GCP Local SSD, Azure ephemeral OS/temp disks) is physically attached flash — the fastest storage available, with millions of IOPS and microsecond latency, and it is included in the instance price. The caveat you must state every time: it is ephemeral. The data is lost on instance stop, on hibernation, and on any hardware failure that migrates the instance — and unlike EBS there is no replication behind it. Correct uses: caches, scratch/spill space for Spark or a database's temp files, and the local storage layer of a distributed system that already replicates across instances (Cassandra, Kafka, ClickHouse, Elasticsearch) — where the system's own replication provides the durability the disk does not. The classic mistake: running a single-node PostgreSQL on local NVMe because it benchmarks beautifully, then discovering that "stopped instance" means "empty disk."

15.6Family 6 — Table formats over object storage#

Covered in full in §12.3: Apache Iceberg (engine-neutral default, snapshot/manifest architecture, best vendor support), Delta Lake (Databricks/Spark-native, transaction-log architecture, best MERGE ergonomics), Apache Hudi (streaming upserts and incremental CDC queries, primary-key index). They are listed here because they are the layer that turns an object store into a database-like table, and no modern data-lake design should omit one.

15.7The comparison table#

15.8Decision rules — say these out loud#

15.9Anti-recommendations — the common wrong picks#

9Real-World Engineering Scenarios#

9.1Amazon S3 — The Reference Object Store#

9.2Google — GFS → Colossus (removing the single master)#

9.3Facebook — Haystack & f4 (photos at social scale)#

10Interview Gotchas & Failure Modes#

10.1Classic Questions#

10.2Failure Modes & Mitigations#

11Whiteboard Cheat Sheet#

 RULE: blob -> object storage ; metadata + URL -> database ; hot reads -> CDN in front

 ABSTRACTIONS:  BLOCK (raw disk: DBs/VMs, mutable, fastest) | FILE (POSIX hierarchy: shared/legacy)
                OBJECT (flat HTTP key->blob: media/backups/data-lake, immutable, near-infinite)

 GFS/HDFS ARCHITECTURE (control/data plane split):
    MASTER/NameNode  ── metadata ONLY (namespace, file->chunks, chunk->locations) in RAM
        │  (client asks master WHERE, then talks to storage nodes directly for BYTES)
        ▼
    CHUNKSERVERS/DataNodes ── store 64-128MB CHUNKS, each replicated x3 (rack/AZ-aware)
    WRITE: master grants PRIMARY lease -> data PIPELINED to replicas -> primary orders -> secondaries apply
    HEAL: heartbeats -> detect dead -> re-replicate to restore R ; checksums + scrub -> catch bit rot
    MASTER SPOF/ceiling -> big chunks(small meta) + data bypasses master + HA(WAL+consensus) + sharded meta

 WAL: append change to durable log BEFORE apply; crash -> replay WAL + checkpoint = exact state (universal)

 DURABILITY:  3x REPLICATION (simple, fast recovery, 200% overhead, tolerate 2)
              ERASURE CODING RS(k,m): any k of k+m recovers; RS(10,4) = 1.4x overhead, tolerate 4
              hot->replicate (fast recovery) ; cold->erasure-code (cheap). Durable != Available.

 OBJECT (S3): buckets + flat keys + immutable objects + versioning + multipart upload + tiers(hot/cold)
              + strongly-consistent metadata index (sharded+replicated) + presigned URLs + CDN origin

One-sentence summary for an interviewer:

"Distributed storage assumes hardware fails constantly, so it chunks files across thousands of nodes and keeps them safe with 3× replication (hot data, cheap recovery) or erasure coding (cold data, ~1.4× overhead for more fault tolerance), all coordinated by a metadata/control plane separated from the data plane so bulk bytes stream directly between clients and storage nodes; you store blobs in object storage (S3) and only their metadata+URL in the database, front it with a CDN for global reads, make the metadata master HA via a replicated WAL + consensus, and remember durability (never lost, 11 nines) is not availability (reachable now)."

End of File 09. Next → File 10: Microservices & Service Mesh (service discovery, circuit breakers, Sagas) — the capstone that ties all 9 prior concepts together.