Concept 7: Message Queues & Event Streaming

System Design Bible — File 07 of 10 Difficulty: ★★★★☆ Prerequisites: Files 01–06, plus the sync-vs-async model in Section 0.

Table of Contents#

0Prerequisites — What You Must Understand First#

Queues exist to fix the problems of synchronous communication. So you must first feel, concretely, what synchronous means and why it hurts at scale.

0.1Synchronous vs asynchronous communication (the core dichotomy)#

Synchronous communication is a blocking call-and-wait: service A calls service B and stops, waiting, until B responds — like a phone call, where you can't do anything else until the other person answers and finishes. In code it's a normal function/HTTP call: result = paymentService.charge(order) — the line doesn't proceed until the charge returns. Asynchronous communication is non-blocking send-and-continue: A hands off a message and immediately moves on, without waiting for the work to be done — like sending a text or dropping a letter in a mailbox. The recipient handles it whenever they can, and A isn't held up. This single distinction is the whole subject of this file: message queues are the machinery for turning synchronous, tightly-coupled service calls into asynchronous, loosely-coupled message passing. Fix this contrast firmly: synchronous = wait for the answer now (phone call); asynchronous = hand it off and continue (mailbox).

0.2Why synchronous coupling is dangerous at scale (temporal + load coupling)#

Synchronous calls create two hidden dependencies that quietly wreck systems. Temporal coupling: because A waits for B, A and B must both be healthy at the same instant. If B is down, A's call fails or hangs — and if A's caller was also waiting on A, that hangs too, and so on: a single failing service can cascade into a system-wide outage, because everyone upstream is blocked waiting (this is the cascading failure of File 10). Load coupling: because A calls B directly and in lockstep, if A suddenly fires 10,000 requests/sec, B must handle 10,000/sec right now or fall over — there's no buffer, no shock absorber, so a spike in one service instantly slams the next. A message queue breaks both: it's a durable buffer between A and B, so A can send even when B is down (B processes later — no temporal coupling) and A can dump a spike into the queue that B drains at its own steady pace (no load coupling). Understanding these two couplings is understanding why queues matter.

0.3What "durable" means and why it's the queue's superpower#

A message store is durable if a message, once accepted, survives a crash — because it was written to disk (and usually replicated), not just held in volatile RAM. This is the property that makes a queue trustworthy as a buffer between services. Picture the alternative: if the queue held messages only in memory and the queue server rebooted, every in-flight message — every un-processed order, payment event, or task — would vanish. Durability means A can hand off a critical message and know it won't be lost even if B is down for an hour and the queue server restarts twice in between; the message waits safely on disk until B is ready. Recall from File 11 (or intuit) that appending to a disk log is fast (sequential writes), which is why durable queues can be both safe and high-throughput. When you see "Kafka persists everything to disk," this is why it's a feature, not a slowdown. Hold it: durable = written to disk, survives crashes = safe to hand off and forget.

0.4What "coupling" actually means — the three couplings, defined separately#

The word coupling gets thrown around as if it were one thing. It is three things, and a message broker attacks each of them by a different mechanism. If you can name all three in an interview and say which one a queue fixes and which one it does not, you sound like someone who has operated these systems rather than read about them.

Coupling, in general, is the degree to which two components must know about, agree with, or depend on each other in order to work. The opposite, decoupling, means each can change, fail, restart, move, or scale without the other noticing. Coupling is not evil — a system with zero coupling does nothing — but hidden coupling is, because it makes the failure of one component silently become the failure of another.

1. Temporal coupling — "we must both be alive at the same instant." This is the one people mean most often. In a synchronous HTTP call, the caller checkout-service and the callee inventory-service must be simultaneously available: checkout is blocked inside the call, holding a thread, a socket, and (usually) an open database transaction, until inventory answers. If inventory is down for 90 seconds during a deploy, every request that arrives in those 90 seconds fails, because there is nowhere for the work to wait. A broker removes temporal coupling by giving the work somewhere to wait: checkout writes a message and returns; inventory reads it whenever it comes back. The producer and the consumer are never required to be up at the same moment — the broker is the only thing that must be up, and brokers are replicated precisely so that they are (§8).

2. Spatial coupling (also called location or addressing coupling) — "I must know exactly who you are and where you live." In a synchronous call, checkout holds a concrete address: a DNS name like inventory.prod.svc.cluster.local, a port, a client library, a protocol version, and often a hard-coded list of the specific downstreams it must notify. That is spatial coupling, and its symptom is the "add a ninth consumer, redeploy the producer" problem: when a new team wants to react to an order being placed, someone must open checkout's source, add a call, re-test it, and re-deploy it — a change to a critical payment path caused by a feature that has nothing to do with payments. A broker removes spatial coupling by replacing "call these eight named services" with "publish one order.placed event to this topic." The producer now addresses a topic — a named channel — not a peer. It does not know how many consumers exist, where they run, what language they are written in, or whether there are zero of them. Adding a ninth consumer is a deployment in the consumer's repository and touches nothing upstream.

3. Failure coupling (also called behavioural or fate coupling) — "your failure becomes my failure." This is the subtlest and the most dangerous. Even when both services are up, a synchronous call couples their behaviour: if inventory gets slow — not down, just slow, p99 climbing from 20 ms to 4 seconds — then checkout's threads pile up waiting, its thread pool exhausts, and checkout starts failing requests that have nothing to do with inventory at all. Slowness has propagated across a service boundary and turned into unavailability. This is the mechanism behind cascading failure, and it is exactly what circuit breakers and bulkheads (File 10) and rate limiters (File 03) exist to interrupt. A broker breaks failure coupling because the producer's write to the broker is fast and bounded (a few milliseconds to a local, replicated, append-only log) and completely independent of how slow the consumer is. A consumer that slows down grows a backlog — a number on a dashboard — instead of setting fire to its callers.

What a broker does NOT decouple, and say this out loud in an interview: it does not decouple you from correctness dependencies. If the user must see the result before the page renders, an async hand-off has not removed the dependency; it has moved it into a polling loop or a websocket push, and you now owe the user an eventual-consistency story (File 19). Queues convert a latency and availability problem into a consistency and observability problem. That is a trade, not a free win (Rule 5).

The arithmetic that makes this concrete — availability multiplication. Availability is the fraction of time a component correctly serves requests, usually quoted in "nines": 99.9% ("three nines") means the service is unavailable at most 0.1% of the time, which is 43.2 minutes per 30-day month; 99.99% ("four nines") is 4.32 minutes per month; 99.999% ("five nines") is 26 seconds per month. Now consider a synchronous call chain of five services — the API gateway calls orders, which calls pricing, which calls inventory, which calls payments — where every one of them is independently a very respectable 99.9% available. Because the request needs all five to be up simultaneously, the availabilities multiply:

  0.999 × 0.999 × 0.999 × 0.999 × 0.999 = 0.99500...  ≈ 99.5%

Your chain is 99.5%, not 99.9%. In minutes per 30-day month: 0.005 × 43,200 minutes = 216 minutes ≈ 3.6 hours of downtime per month, up from 43 minutes. You did not build anything unreliable; you simply put five reliable things in series. Add five more services and you are at 99.0% — 7.2 hours a month. Serial synchronous dependencies multiply failure probability, and the multiplication is brutal. The general form: for n services each with availability a, the chain's availability is aⁿ, so the unavailability is roughly n × (1 − a) for small failure rates — five services at 0.1% failure each ≈ 0.5% chain failure. Adding a service to a synchronous path adds its entire failure budget to yours.

Latency adds the same way, but worse, because tails compound. If each of those five services has a median latency of 20 ms, the chain's median is about 100 ms — that part is simple addition. But the tail is not additive in the friendly way: if each service independently has a p99 of 500 ms (meaning 1 request in 100 is that slow), then the probability that a five-hop request avoids every slow hop is 0.99⁵ ≈ 0.951 — so roughly 5% of your requests hit at least one 500 ms hop, and your chain's p95 is now around half a second. This is the phenomenon Google's Jeff Dean named "the tail at scale": the more components a request must touch, the more likely it is to touch a slow one, so tail latency degrades far faster than median latency as you add hops.

The asynchronous alternative changes the shape of the arithmetic. When checkout writes to a broker and returns, the user-facing availability depends on checkout and the broker only — two components, 0.999 × 0.9999 (brokers are replicated, so they are usually the most available thing you own) ≈ 99.89%. The other four services' outages no longer show up as user-visible errors; they show up as consumer lag (§7.1), a backlog that drains when they recover. You have not made payments more reliable. You have made its unreliability invisible to the checkout path and recoverable without data loss. That is the entire value proposition of this file in one sentence.

Hold this: temporal coupling = must be up together; spatial coupling = must know each other's address; failure coupling = your latency becomes my outage. A broker breaks all three, at the price of eventual consistency and a backlog you must monitor.

0.5Durability for real — the page cache, fsync, and why "written" does not mean "durable"#

§0.3 said durable means "written to disk." That sentence hides the single most misunderstood mechanism in storage systems, and every broker's durability configuration is a knob on top of it. You cannot reason about acks=all, about flush.messages, or about "did we lose data when that broker lost power?" without knowing what actually happens when a program calls write().

Where the bytes actually go. When your process calls write(fd, buffer, 4096) on Linux, the kernel does not put those bytes on the physical disk. It copies them into the page cache — a region of ordinary RAM the kernel uses to hold copies of file data, organised in 4 KB pages — marks those pages dirty (meaning "this RAM copy is newer than the disk copy"), and returns success to your program, typically in a few microseconds. Your program now believes the data is written. It is in volatile memory. If the machine loses power one millisecond later, those bytes are gone, and the file on disk looks as if the write never happened.

A kernel thread (historically pdflush, today the per-device writeback workers) walks the dirty pages in the background and issues the real disk I/O, driven by two thresholds you can inspect on any Linux box: **vm.dirty_background_ratio (default around 10 — start writing back in the background once dirty pages exceed 10% of available memory), vm.dirty_ratio** (default around 20 — once dirty pages exceed 20%, block the writing process until writeback catches up, which is why a machine doing heavy writes can suddenly stall), and **vm.dirty_expire_centisecs (default 3000 — write back any page dirty for more than 30 seconds). So in the default configuration, a written byte may sit in RAM, unwritten, for up to 30 seconds.**

**Sub-Concept: fsync.** fsync(fd) is the system call that says: do not return until every dirty page belonging to this file, and the file's metadata, has been handed to the storage device and the device has confirmed it is in non-volatile storage. It is the only way a program can convert "written" into "durable." Its sibling fdatasync(fd) is the same thing but skips metadata that does not affect reading the data back (like the file's modification timestamp), which saves one extra I/O and is what most databases and brokers actually call. There is also O_DIRECT, an open-time flag that bypasses the page cache entirely and does the I/O straight from your buffer to the device — used by databases that want to manage their own cache and refuse to have data double-buffered, but it forfeits the free read caching the page cache provides, which is why Kafka deliberately does not use it (§4.5).

What it costs, with real numbers, because the number is the whole story. An fsync is a round trip to the storage device, and the device must actually commit the data before answering: - Spinning disk (HDD, 7,200 rpm): the platter must rotate to the right position. Average rotational latency is half a revolution = 4.2 ms, plus seek time of ~5–9 ms if the head must move. A single fsync costs roughly 5–15 ms. That caps you at ~70–200 durable commits per second per disk. This is why every database in the 1990s batched commits. - SATA SSD: no moving parts, but the flash translation layer must program a page and update its mapping. fsync costs roughly 0.5–2 ms — call it 500–2,000 durable commits/sec. - NVMe SSD (PCIe-attached, datacenter class): the command path is much shorter and the device has deep parallelism. fsync costs roughly 20–200 microseconds — 5,000–50,000 durable commits/sec, and far more if you have many files in flight. - Battery-backed / power-loss-protected write cache: enterprise RAID controllers and datacenter SSDs contain a small DRAM write buffer plus either a BBU (battery backup unit) or a bank of supercapacitors holding enough charge to flush that buffer to flash after a power cut. Because the buffer survives power loss, the device can honestly answer "committed" the instant the bytes land in DRAM. fsync then costs tens of microseconds even on spinning disks behind the controller. This is the single hardware feature that most changes a storage system's throughput, and it is why "the same software" runs 50× faster on a datacenter SSD than on the consumer SSD in a laptop — consumer SSDs usually have no power-loss protection and either lie about flushing or are genuinely slow at it.

What breaks. Three classic failure modes. (a) The lying disk: some consumer drives acknowledge a cache-flush command without actually flushing, so fsync returns and the data is still volatile — you get corruption only on real power loss, which is why it is discovered in production and never in testing. You can disable the volatile write cache with hdparm -W0 /dev/sda at a throughput cost. (b) **The fsync error-swallowing bug ("fsyncgate", 2018):** on Linux, if a writeback error occurred, older kernels reported the error to the first fsync that noticed and then cleared the flag, so a second fsync returned success even though the data was never written — PostgreSQL had to change its recovery strategy to panic-and-replay-WAL on any fsync failure rather than retry. The lesson: **an fsync that returns an error means your process's understanding of what is on disk is unrecoverable; the only safe response is to crash and replay the log. (c) The fsync stall:** because fsync blocks until the device answers, a single slow device turns into seconds of application latency, and if your broker fsyncs on the request path, that latency is your producer's latency. iostat -x 1 showing high await and %util is how you see it.

Now the point for brokers. Every broker chooses a position on this spectrum, and the choice is the durability/throughput trade:

Hold this: **write() puts bytes in volatile RAM and returns; only fsync makes them durable, and it costs 5–15 ms on a spinning disk, ~1 ms on a SATA SSD, ~50 µs on NVMe, and ~50 µs on anything with a battery-backed cache. Every broker's durability configuration is a decision about how often to pay that cost — or whether to buy durability with replication instead.**

0.6Sequential vs random I/O — why an append-only log is absurdly fast#

The reason a log-structured broker can do a million messages a second on hardware that a traditional database would choke on is not clever code. It is a property of physical storage devices that has held for forty years: sequential access is one to three orders of magnitude faster than random access, and an append-only log is the most purely sequential access pattern that exists.

What the two terms mean precisely. Sequential I/O means reading or writing bytes at consecutive addresses — you write block 1000, then 1001, then 1002. Random I/O means each operation touches an unrelated address — write block 91,442, then block 12, then block 700,301. On a spinning disk the difference is physical and obvious: sequential writes let the head stay put while the platter spins underneath it, so you pay one seek and then stream; random writes pay a seek (moving the head arm, 4–9 ms) plus rotational latency (waiting for the right sector to come around, average 4.2 ms at 7,200 rpm) for every single operation. On an SSD there is no head, but the difference survives for a different reason: the flash translation layer, erase-block granularity (flash can be written in 4–16 KB pages but only erased in 1–4 MB blocks), and write amplification (a small random write forces the controller to read, modify, and rewrite a whole erase block, and eventually to garbage-collect) mean random writes cost more device work and wear out the drive faster.

The numbers, which you should be able to quote:

Read the HDD row again, because it is the row that shaped this entire class of software: a 7,200 rpm disk does about 150 MB/s sequentially and about 0.5 MB/s in random 4 KB operations. That is a factor of ~300. The often-quoted line from the Kafka design documents is exactly this comparison: sequential writes to a commodity disk array ran at ~600 MB/s while random writes on the same hardware ran at ~100 KB/s. Sequential disk is faster than random RAM access in some benchmarks — not because disks are fast, but because random memory access misses the CPU cache and random disk access misses everything.

Why an append-only log is the ideal shape. A log here means a file you only ever add to at the end — never update in place, never delete from the middle. Every write is lseek(END); write(), which on the physical device is "the next block after the last one." So:

The cost side, because nothing is free (Rule 5). A log gives up random access: to find "the message with ID X" you must either scan or maintain a separate index. Kafka's answer is a sparse .index file per segment mapping offset → byte position roughly every 4 KB, so a lookup is a binary search in a small file plus a short scan — good enough for "resume at offset 481,003," useless for "find all messages where user_id = 42." A log also cannot update in place, so correcting data means appending a correction (or compacting, §9.2), and it stores everything, so it costs disk proportional to throughput × retention (§9.4). You buy enormous write throughput and simple recovery by giving up random reads and in-place updates. That is the same bargain LSM-trees make in databases (File 11) and it is not a coincidence — Kafka is essentially the write-ahead log of a database, exposed as a product.

Hold this: sequential ≫ random (300× on disk, ~2× even on NVMe), an append-only log makes every write sequential and every tail-read cache-resident, and that single structural choice is why a log broker outruns a queue broker by an order of magnitude.

0.7The cast of characters — producer, consumer, broker, topic, partition, offset#

Before any mechanism, fix the nouns. Every one of these gets a fuller treatment later; this is the map so that nothing in Section 1 is a word you have not met.

Producer (or publisher). Any process that sends a message into the broker. It is a role, not a product: your checkout web server is a producer when it emits order.placed; a Debezium connector reading a MySQL binlog is a producer; a stream processor writing its output is a producer. In practice a producer is a client library embedded in your application (KafkaProducer in Java, confluent-kafka in Python, amqplib in Node) that maintains connections to the brokers, batches records, compresses them, and retries on failure. The important consequence of it being a library rather than a sidecar is that its configuration — batch size, retries, acks — lives in your application's config and is your team's responsibility.

Consumer (or subscriber). Any process that reads messages out of the broker and does something with them. Also a library. The critical distinction that this whole file rests on: in a queue, the broker pushes or hands out a message and then tracks whether that consumer finished; in a log, the consumer asks for "bytes starting at offset N," gets them, and is responsible for remembering where it got to. That difference in who holds the bookmark determines almost every other difference between RabbitMQ and Kafka.

Broker. A server process that accepts messages, stores them, and serves them to consumers. A production deployment is a cluster of brokers — typically 3, 5, or dozens — that divide the data among themselves and replicate it to each other so that losing one machine loses nothing. "Kafka" in a sentence like "we put it in Kafka" means "the cluster of broker processes plus their disks."

Message (or record, or event). The unit of data. It has a payload (the bytes you care about — a serialized order, a click, a sensor reading), and metadata: usually a key (used for partitioning and compaction), a timestamp, and headers (arbitrary key–value pairs used for things like a trace ID, a schema version, or a retry counter). "Message" and "event" are often used interchangeably; a mild convention is that a command/message is addressed to a specific handler and expects work ("charge this card"), while an event is a statement of fact broadcast to whoever cares ("a card was charged"). The distinction matters for design because commands have one owner and events have none.

Topic. A named, durable category of messages — orders.placed, clicks.raw, payments.settled. Producers write to a topic; consumers subscribe to a topic. It is the unit of addressing that replaces the hard-coded service name and therefore the thing that kills spatial coupling (§0.4). In RabbitMQ the equivalent naming layer is an exchange plus routing key (§14.2.1), which is more flexible per message but the same idea.

Partition. A topic is physically split into partitions, each of which is one independent append-only log file sequence on one broker's disk (replicated to others). Partitions exist for exactly two reasons and you should be able to state both: parallelism (different partitions live on different machines and are read by different consumers simultaneously, so throughput scales with partition count) and ordering (order is guaranteed within a partition and nowhere else, so partitioning is how you decide what must stay ordered relative to what). This is File 04's sharding applied to a log, and the partition key is a shard key with all the same hot-key dangers (File 06).

Sub-Concept: the offset (a cursor). An offset is a 64-bit integer naming a position in one partition's log: the first message ever written to that partition is offset 0, the next is 1, the next 2, forever, monotonically increasing and never reused. It is not a byte position and not a global ID — offset 57 exists independently in every partition of every topic, and offset 57 in partition 0 has nothing to do with offset 57 in partition 1. Think of the partition as a book whose pages are numbered from 1 to infinity, and the offset as a bookmark.

Why it exists. The alternative — the broker remembering, per message per consumer, whether that consumer has finished it — costs the broker a mutable per-message record and a random write for every acknowledgement, which is exactly the random-I/O pattern §0.6 says is 300× slower. Replacing "N pieces of per-message state" with "one integer per consumer per partition" is the single change that lets the broker be dumb and fast: to serve a consumer, the broker does seek(offset) ; sendfile(bytes) and nothing else.

Where it sits. In Kafka the offset is assigned by the partition leader at append time and is implicit in the log's physical layout; the mapping from offset to byte position lives in a per-segment sparse index file (00000000000000000000.index). The consumer's committed offset — its saved bookmark — lives in a special internal compacted topic called __consumer_offsets, keyed by (group.id, topic, partition). The important consequence: a consumer's progress is itself just messages in a Kafka topic, replicated and durable like everything else, which is why a consumer can crash and restart on a different machine and resume exactly where it left off.

Step by step, with values. Partition orders-3 currently ends at offset 1,000,417 (this is its log end offset, LEO). Consumer group billing last committed offset 1,000,000. On start-up the consumer asks the broker "give me records from orders-3 starting at 1,000,000, up to 1 MB," gets a batch of, say, 512 records (offsets 1,000,000–1,000,511), processes them, and commits 1,000,512 — note that a committed offset is the next offset to read, not the last one processed, which is a genuine source of off-by-one bugs when people write their own offset management. Its lag is 1,000,417 − 1,000,512 → it has caught up. If the consumer instead crashes after processing 300 records and before committing, it restarts at 1,000,000 and reprocesses all 300 — which is where at-least-once duplicates come from (§5.2).

Every named thing on this path, individually. auto.offset.reset decides what happens when a group has no committed offset (a brand-new group, or one whose committed offset has aged out of retention): earliest means start from the oldest message still retained (replay everything — correct for analytics, catastrophic if you accidentally reset a payments consumer), latest means start from the end and ignore all history (correct for live dashboards, silently loses everything produced while the consumer was down), and none means throw an exception and make a human decide (the safest for anything financial). enable.auto.commit=true makes the client commit the offsets of everything returned by the previous poll() on a timer (auto.commit.interval.ms, default 5000) — convenient and the single most common cause of message loss, because the commit can happen before your processing finishes. Setting enable.auto.commit=false and calling commitSync() after your work is done is the at-least-once discipline (§5.2).

What breaks. (a) Offset out of range: your committed offset is older than the oldest retained message because your consumer was down longer than the retention period — the data you never read is gone, and auto.offset.reset silently decides whether you skip it or replay from the beginning. Diagnose with kafka-consumer-groups.sh --describe --group billing, which prints CURRENT-OFFSET, LOG-END-OFFSET, and LAG per partition. (b) Committing before processing → loss on crash. (c) Committing an offset you did not fully process because you used auto-commit with a long-running handler.

Tie-back: the offset is the entire reason a log broker can serve thousands of independent consumers of the same data without storing per-consumer, per-message state — and the reason you can replay history by simply writing a smaller number into a bookmark.

Consumer group. A named set of consumer processes that share the work of one subscription — the mechanism that lets a log deliver both queue semantics (within a group, each partition goes to exactly one member, so each message is handled once) and pub/sub semantics (across groups, everyone gets a full copy). Fully treated in §6.

Hold this: producers write to topics, topics are split into partitions, partitions are append-only logs of offset-numbered messages, consumers read forward by offset and remember their own bookmark, and consumer groups are how many consumers share one subscription.

1Architectural Definition & "The Why"#

1.1The Plain Definition#

A message queue (broadly, a message broker) is an intermediary component that lets services communicate asynchronously by passing messages through it, rather than calling each other directly. A producer writes a message to the broker; the broker stores it durably; a consumer reads it later, on its own schedule. Producer and consumer are decoupled — they don't need to be online at the same time, run at the same speed, or even know about each other.

Event streaming (e.g., Apache Kafka) is a related but distinct paradigm: an append-only, durable, replayable log of events that many consumers can read independently and re-read from any point. (The queue-vs-log distinction is central — Section 3.)

1.2"The Why" — Synchronous Coupling Doesn't Scale#

Early service architectures made synchronous calls: service A calls service B over HTTP and blocks, waiting for B's response. This creates crippling problems as systems grow:

The message queue solves all four by inserting a durable buffer between services:

This is the backbone of event-driven architecture and microservices (File 10). Queues are how you turn a brittle web of synchronous calls into a resilient, elastic, loosely-coupled system.

2The Recursive Sub-Concept Tree#

2.1Sub-Concept: Producer, Consumer, Broker, Message#

2.2Sub-Concept: Point-to-Point (Queue) vs. Publish/Subscribe (Topic)#

Kafka elegantly supports both: within a consumer group, each message goes to one member (queue semantics); across different consumer groups, each group gets its own copy (pub/sub semantics). One system, both patterns.

2.3Sub-Concept: Acknowledgement (ACK) & Offset#

2.4Sub-Concept: Durability & Persistence#

A message is durable if it survives a broker crash — i.e., it's written to disk (and usually replicated to other brokers) before the producer is told "accepted." Non-durable/in-memory brokers are faster but lose messages on crash. Durability is the difference between "I might lose your payment event" and "I won't." Kafka persists everything to disk by design; it's a log, not a transient buffer.

2.5Sub-Concept: Idempotency (the consumer's survival kit)#

Idempotency means processing the same message twice has the same effect as processing it once. Because most systems deliver at-least-once (duplicates happen — Section 5), consumers must be idempotent to be correct. Techniques: dedupe on a unique message/event ID (store processed IDs), use upserts instead of inserts, make operations naturally idempotent (set balance = 100 not add 10). "At-least-once delivery + idempotent consumers = effectively-once processing" is one of the most important sentences in distributed systems.

2.6Sub-Concept: Consumer Group#

A consumer group is a set of consumer instances that cooperate to consume a topic, with each partition assigned to exactly one consumer in the group. It's how you scale out consumption: add consumers to the group to process more partitions in parallel. Kafka rebalances partition assignments when members join/leave.

The full mechanism — how the group is coordinated, which algorithm assigns which partition to which member, what a rebalance actually stops, and how it goes wrong — is §6. Read that section before an interview; "consumer group" is the most common follow-up question after "what is a partition."

2.7Sub-Concept: The Partition — the unit of parallelism, ordering, and storage#

Plain definition. A partition is one independent, ordered, append-only log. A topic is nothing but a name for a set of partitions; all the actual data lives in them.

Why it exists. A single log file lives on a single machine, and a single machine has a finite disk, a finite NIC (File 01 §0.6 — 10 Gbps is 1.25 GB/s and that is a hard wall), and a single writer. If a topic were one log, its ceiling would be one machine's ceiling. Partitioning splits the topic across machines so that throughput scales horizontally: 100 partitions spread over 10 brokers give you ten machines' worth of disk, network, and CPU for one logical stream. Partitioning is sharding (File 04) with the shard being a log rather than a table, and the partition key is a shard key (File 06's consistent-hashing lessons about key skew apply directly).

Where it sits — the physical layout. On a Kafka broker's disk, a partition is a directory named <topic>-<partition-number>, e.g. orders-3, inside the path given by log.dirs. Inside that directory are segment files:

/var/lib/kafka/orders-3/
    00000000000000000000.log     <- messages, offsets 0 .. 1,048,575
    00000000000000000000.index   <- sparse map: offset -> byte position in .log
    00000000000000000000.timeindex <- sparse map: timestamp -> offset
    00000000000001048576.log     <- ACTIVE segment, currently being appended
    00000000000001048576.index
    00000000000001048576.timeindex
    leader-epoch-checkpoint      <- which broker led during which epoch (§8.5)

The number in each filename is the base offset — the offset of the first message in that segment. Only the last segment (the active segment) is written to; the rest are immutable and are the unit of deletion and compaction (§9).

How it works, step by step, with values. A producer sends a record with key user-42 to topic orders, which has 6 partitions.

Every named thing, individually. num.partitions is the broker-level default partition count for auto-created topics (default 1 — almost never what you want). partitions is the per-topic setting given at creation with kafka-topics.sh --create --partitions 12. You can increase a topic's partition count but never decrease it (kafka-topics.sh --alter --partitions 24), and increasing it **changes the hash mod N mapping for every key**, so user-42 may move from partition 3 to partition 15 — meaning its old events are in one partition and its new ones in another, and per-key ordering is broken across the boundary forever. That is the single most important operational fact about partition counts, and it is why the standard advice is to over-provision partitions at creation (pick 2–4× your expected consumer count) rather than plan to grow. max.message.bytes (broker/topic, default 1,048,588 ≈ 1 MB) caps one record batch; exceed it and the producer gets RecordTooLargeException, which is what drives the claim-check pattern (§11.4).

What breaks. (a) Too few partitions caps consumer parallelism — with 4 partitions you can never usefully run more than 4 consumers in a group, no matter how far behind you are, so your only recovery from lag is to make each consumer faster. (b) Too many partitions costs real resources: each partition is at least 3 open file handles per replica, a separate batch buffer in every producer (so 10,000 partitions × 16 KB = 160 MB of producer heap), a separate fetch request stream, and — critically — leader-election time after a broker failure scales with partition count, so a cluster with 200,000 partitions can take minutes to recover. Modern KRaft-based Kafka handles millions of partitions far better than the ZooKeeper era did, but the rule of thumb "a few thousand partitions per broker, tens of thousands per cluster" is still sound. (c) Skewed keys put most traffic on one partition (§5.7). Diagnose partition-level throughput with kafka-log-dirs.sh --describe for size and JMX BytesInPerSec per topic-partition for rate.

Tie-back. Every guarantee this file discusses is scoped to a partition: ordering is per-partition, an offset is per-partition, a consumer's assignment is per-partition, replication is per-partition. "Partition" is the noun in every precise sentence about Kafka.

2.8Sub-Concept: Push vs. Pull delivery#

Plain definition. In a push model the broker takes the initiative and sends messages to a connected consumer as soon as they arrive. In a pull (or poll) model the consumer takes the initiative and asks the broker "give me messages from position X."

Why the distinction exists. It is a control question: who decides the rate? If the broker decides (push), the broker can outrun the consumer, and something must stop it — otherwise the consumer's in-memory buffer grows until the process dies of OOM (out of memory: the allocator cannot satisfy a request, and on Linux the kernel's OOM killer terminates the process). If the consumer decides (pull), the rate is trivially bounded by the consumer's own loop, and the backlog accumulates on the broker's disk, which is cheap and durable. This is the mechanism behind §7.1's claim that Kafka gets backpressure for free.

How each works, concretely. RabbitMQ pushes: a consumer issues basic.consume on a queue and the broker streams deliveries down the channel as messages arrive, without further requests. To stop the broker outrunning the consumer, RabbitMQ uses prefetch (the basic.qos(prefetch_count=N) setting, also called QoS): the broker will have at most N unacknowledged messages outstanding to that consumer at a time, and sends the N+1st only when one is acked. prefetch_count=1 gives perfect fairness and lowest memory but a round trip of idle time per message; prefetch_count=100 gives high throughput but means a crashed consumer redelivers up to 100 messages and a slow consumer can hoard 100 messages that a free peer could have handled. The tuning rule: prefetch ≈ (round-trip time ÷ processing time) rounded up, so a 1 ms RTT and 10 ms of processing wants prefetch 1–2, while a 1 ms RTT and 0.1 ms of processing wants prefetch ~50.

Kafka pulls: the consumer calls poll(timeout) in a loop, which issues a Fetch request naming, per partition, the offset to start at and how many bytes it wants. fetch.min.bytes (default 1) tells the broker "do not answer until you have at least this many bytes," and fetch.max.wait.ms (default 500) caps how long it will hold the request waiting for them — together these implement long polling, which avoids the naive pull model's flaw of hammering the broker with empty requests when the topic is idle. max.partition.fetch.bytes (default 1,048,576) caps bytes per partition per fetch; max.poll.records (default 500) caps how many records one poll() hands your code, which is the knob that keeps you inside max.poll.interval.ms (§6.5).

Pros and cons of each. Push wins on latency — the message goes out the instant it arrives, with no polling interval — and it lets a smart broker do per-message routing and per-consumer fairness. Push loses on flow control (needs an explicit credit mechanism), on batching (the broker doesn't know how much the consumer wants), and on replay (the broker is driving, so "go back and re-read" is not a natural operation). Pull wins on backpressure (automatic), batching (the consumer asks for a megabyte and gets a megabyte, which is what makes zero-copy and compression pay off), and replay (just ask for an older offset). Pull loses on idle latency unless you implement long polling, and it pushes bookkeeping onto the consumer.

When to use which. Choose push when per-message latency is the product (chat delivery, trading signals) and message volume per consumer is modest. Choose pull when throughput and backlog tolerance matter more than the last few milliseconds — which is nearly all data-pipeline work.

What breaks. In push systems, the classic incident is prefetch too high + slow consumer: one consumer holds hundreds of unacked messages, the broker's queue looks drained but nothing is progressing, and if that consumer dies all of them redeliver at once. Diagnose in RabbitMQ with rabbitmqctl list_queues name messages_ready messages_unacknowledged — a large messages_unacknowledged with a small messages_ready is exactly this. In pull systems, the classic incident is a consumer whose processing loop takes longer than max.poll.interval.ms and is evicted from the group mid-batch (§6.5).

Tie-back. "Kafka gives you backpressure for free" is not magic; it is the direct consequence of the consumer, not the broker, issuing the fetch.

2.9Sub-Concept: The Topic, the Queue, and the Subscription — three different nouns#

These three words are used almost interchangeably in conversation and mean different things in every product, which is a genuine source of interview confusion.

A queue (RabbitMQ, SQS, ActiveMQ) is a storage object holding messages awaiting delivery. It has an identity, a depth you can measure, and consumers attached to it. Messages leave it when acknowledged. Multiple consumers on one queue compete — each message goes to exactly one of them.

A topic in the traditional pub/sub sense (JMS topics, SNS, Google Pub/Sub, MQTT) is a broadcast channel, and it typically holds no messages itself — it is a routing point. What holds messages is a subscription: each subscriber creates its own subscription to the topic, and the broker copies each published message into every subscription. In Google Pub/Sub this is explicit: you publish to a topic, and each of your three services has its own subscription with its own backlog and its own acknowledgements. In AWS, the equivalent shape is SNS (topic) fanning out into several SQS queues (subscriptions), which is the canonical "SNS→SQS fan-out" pattern (§14.3.2).

A topic in Kafka is a different animal wearing the same name: it is the log itself, it does hold the data, there are no per-subscriber copies, and "subscription" is replaced by "a consumer group with a stored offset." One physical copy of the data serves every reader. This is why Kafka's fan-out is free and a traditional pub/sub broker's fan-out is not: adding a tenth consumer group to a Kafka topic adds zero bytes of storage and only some read bandwidth, whereas adding a tenth SNS subscription creates a tenth physical copy of every message.

Tie-back: when an interviewer says "topic," ask which model they mean, or state which you mean. Saying "in Kafka the topic is the storage, so subscribers share one copy; in SNS the topic is a router and each subscription gets its own copy" is a senior-level distinction in one sentence.

2.10Sub-Concept: Visibility timeout and the ack deadline#

Plain definition. A visibility timeout (AWS SQS's name) or ack deadline (Google Pub/Sub's name) is a timer that starts when the broker hands a message to a consumer: for the duration of that timer, the message is hidden from all other consumers; if the consumer acknowledges within the timer, the message is deleted; if the timer expires without an ack, the message becomes visible again and another consumer will get it.

Why it exists. In a queue, the broker owns the per-message state, so it must decide when a message that was handed out but never acknowledged should be considered abandoned. A consumer that crashed mid-processing sends no signal — a dead process cannot report its own death. The only mechanism available is a timeout: assume that "no ack within T seconds" means "the consumer is gone." It is the queue-world equivalent of Kafka's max.poll.interval.ms eviction, and it produces the same trade-off.

Step by step, with values. SQS default visibility timeout is 30 seconds, maximum 12 hours. A worker calls ReceiveMessage, gets message m with a receipt handle (an opaque token identifying this particular delivery, which you must present to delete or extend it — note that a redelivery gets a different receipt handle for the same message). The worker has 30 seconds. If its job takes 45 seconds, then at t=30 SQS re-delivers m to a second worker, both workers finish, both call DeleteMessage, and you have processed the message twice — silently, with no error anywhere. The correct fixes are: set the timeout above your p99.9 processing time, and/or call ChangeMessageVisibility periodically to extend it (a heartbeat), and/or use the SDK's automatic extension feature, and — always — be idempotent (File 15).

Every named thing. VisibilityTimeout — the per-queue default, overridable per ReceiveMessage call. ChangeMessageVisibility(receiptHandle, N) — extend or shorten this one delivery's timer; setting N=0 makes the message immediately visible again, which is the idiomatic way to say "I cannot handle this, give it to someone else now" instead of waiting out the timer. ApproximateReceiveCount — a message attribute the consumer can read to know how many times this message has been delivered, which is how you implement "after 5 attempts, treat as poison" without external state (and it is what SQS's native redrive policy uses: maxReceiveCount on the queue's RedrivePolicy sends a message to the DLQ automatically once the count is exceeded). ApproximateNumberOfMessagesNotVisible — the CloudWatch metric showing in-flight messages, i.e. handed out but not yet acked; SQS caps in-flight messages at 120,000 for a standard queue and 20,000 for a FIFO queue, and hitting that cap makes ReceiveMessage start returning empty, which looks exactly like an empty queue and is a famously confusing incident.

What breaks. Timeout too short → duplicate processing on every slow message, and worse, a feedback loop: duplicates add load, load makes processing slower, slower processing produces more duplicates. Timeout too long → a genuinely crashed consumer's messages are stuck invisible for hours, so a crash converts into latency you cannot see in queue depth. Diagnose with the ratio of NumberOfMessagesReceived to NumberOfMessagesDeleted — significantly more receives than deletes means you are redelivering.

Tie-back. Visibility timeout is why queues are at-least-once: the broker must guess whether an unacknowledged message was processed, and the safe guess — redeliver — creates duplicates (§5.2).

2.11Sub-Concept: Consumer lag#

Plain definition. Consumer lag for a partition is log end offset − consumer's committed offset: the number of messages that have been written and not yet processed by that consumer group. Summed across partitions, it is the size of your backlog.

**Why it is the metric.** Every other symptom of a broken streaming system shows up here first. A consumer that crashed: lag rises linearly at the production rate. A consumer that got slow (a downstream DB degraded, a GC pause, a bad deploy): lag rises at the difference between production and consumption rate. A rebalance storm (§6.4): lag saws up and down. A partition with a poison message (§7.3): one partition's lag rises while the others stay flat — which is why you must alert on per-partition lag, not the sum. A producer spike: lag rises and then drains, which is the system working correctly. Lag is the one number that tells you whether the asynchronous system is keeping its promise, because in an async system there are no user-visible errors to alert on — the errors have been converted into delay.

How to read it, with values. kafka-consumer-groups.sh --bootstrap-server b:9092 --describe --group billing prints per partition: CURRENT-OFFSET (committed), LOG-END-OFFSET (latest produced), LAG (the difference), and CONSUMER-ID. A row showing LAG 4,200,000 on partition 7 and LAG 0 everywhere else is a single stuck consumer or a hot partition, not a capacity problem — adding consumers will not help, because partition 7 can only ever be served by one member of the group.

Lag in messages vs lag in time. 4 million messages of lag is meaningless without a rate. Two derived metrics matter more: time lag = (timestamp of latest produced record − timestamp of the record at the committed offset), which answers "how stale is my output?" in seconds and is what an SLA is written against; and drain time = lag ÷ (consumption rate − production rate), which answers "if nothing changes, when do I catch up?" and is negative when you will never catch up — the number that tells you to scale now. Burrow (LinkedIn's open-source lag monitor) and Cruise Control evaluate lag trend rather than absolute value for exactly this reason: a constant lag of 50,000 on a 500,000/s topic is 0.1 seconds of staleness and completely healthy.

What breaks. Alerting on absolute lag produces pages during every normal traffic spike and misses a slowly-drifting consumer on a low-volume topic. Alerting on lag without also alerting on "consumer group has zero members" misses the case where the consumer is entirely gone and its lag stops being reported. And lag that exceeds retention is not lag any more — it is data loss: the messages you had not read were deleted (§9.1), and the consumer will silently jump forward per auto.offset.reset.

Tie-back. Every mitigation in this file — scale consumers, add partitions, fix the key, raise retention, ship the poison message to a DLQ — is chosen by reading the shape of the lag graph.

2.12Sub-Concept: The Dumb Broker / Smart Consumer inversion#

Plain definition. "Smart broker, dumb consumer" describes a design where the broker does routing, filtering, per-message state tracking, retry scheduling, and delivery decisions, and the consumer just receives and acks. "Dumb broker, smart consumer" describes the inverse: the broker does almost nothing per message beyond appending bytes and serving byte ranges, and the consumer owns position tracking, retry policy, and dispatch.

Why this inversion is the deepest idea in the file. Per-message broker work is random, mutable, small-write work — exactly the pattern §0.6 showed is 300× slower on disk. A smart broker must, for every message, hold state ("delivered to consumer 7 at 12:00:03, unacked"), mutate that state on ack, and schedule a timer for redelivery. That is a database transaction per message. A dumb broker's per-message cost is appending bytes to a buffer — amortised to almost nothing because thousands of messages are appended in one sequential write and served with one sendfile call. The throughput gap between RabbitMQ (tens of thousands of messages/sec per node in durable mode) and Kafka (hundreds of thousands to millions per node) is almost entirely this design choice, not implementation quality.

What you give up by making the broker dumb. Everything that requires the broker to understand individual messages: per-message TTL, priority queues, delayed delivery ("deliver this at 3 p.m."), selective consumption ("give me only messages where type = refund"), per-message routing across many destinations, and the ability to not deliver a message that has already been handled. Kafka can only ever say "here are the bytes from offset N onward" — if a consumer wants only 1% of them, it reads 100% and discards 99%. Those features are exactly RabbitMQ's and SQS's reason to exist, and "we need per-message delay and priority" is a completely legitimate reason to choose a traditional broker over a log (§14.6).

Tie-back. Every row of the comparison table in §3.3 is a consequence of this one inversion. If you can only remember one framing from this file, remember this and the queue/pub-sub/log trichotomy of §3.0.

3Queue vs. Log — The Two Paradigms#

This distinction separates junior from senior answers. There are two fundamentally different broker designs.

3.1Traditional Message Queue (RabbitMQ, ActiveMQ, SQS)#

Model: Messages are consumed and deleted. A message sits in the queue; a consumer takes it, processes it, ACKs it, and it's removed. The queue is a transient buffer — once delivered and acked, the message is gone.

3.2Event Streaming Log (Kafka, Pulsar, Kinesis)#

Model: Messages are appended to an immutable, ordered log and retained (for days, or forever) regardless of consumption. Consumers read by offset and do not delete anything — many consumers read the same log independently, each tracking its own position.

3.3The Core Distinction Table#

Interview line: "A queue is a buffer you drain; a log is a source of truth you replay." Choose the queue for jobs, the log for events.

4Deep-Dive Mechanics: Apache Kafka#

Kafka is the dominant event-streaming system and the most-asked. Know its internals cold.

4.1Topics and Partitions#

4.2Offsets and the Log#

4.3Consumer Groups & Parallelism#

4.4Replication & Durability (how Kafka doesn't lose data)#

4.5Why Kafka Is So Fast (the mechanics interviewers love)#

4.6Log Compaction (the "latest value per key" mode)#

Besides time/size retention, Kafka offers log compaction: retain at least the latest message for each key, garbage-collecting older values of the same key. This turns a topic into a durable, replayable changelog / materialized table — perfect for CDC and for rebuilding state (e.g., a topic of user_id → current_profile). Foundational to Kafka Streams' state stores and event sourcing — the architectural pattern where you store not the current state of an entity but the full ordered sequence of events that produced it (AccountOpened, Deposited $50, Withdrew $20…), and derive current state by replaying the events. The event log becomes the source of truth: you get a perfect audit trail, the ability to reconstruct state at any past moment, and the freedom to build brand-new views by re-reading history — at the cost of eventual consistency in the derived views and more complex querying. Kafka's retained, replayable log is the natural storage for it.

4.7Coordination: ZooKeeper → KRaft#

Sub-Concept: ZooKeeper. ZooKeeper is a distributed coordination service — a small, strongly-consistent, replicated key-value/hierarchical store used for cluster metadata, leader election, and configuration (it uses a consensus protocol, ZAB, cousin of Raft/Paxos — File 08). Historically Kafka used ZooKeeper to track brokers, topic metadata, and elect partition leaders. KRaft (Kafka Raft, since ~2021) replaces ZooKeeper with a built-in Raft-based controller quorum, removing the external dependency and improving scalability. Point: any large distributed system needs a consensus-backed source of truth for membership and leadership — that's File 08.

5Delivery Guarantees & Ordering#

5.1The Three Delivery Semantics#

5.2The Truth About "Exactly-Once"#

Exactly-once delivery across a network is theoretically impossible. The classic proof-sketch is the Two Generals Problem: two generals on separate hills must attack a valley simultaneously to win, and can only coordinate by messengers who may be captured crossing the valley. General A sends "attack at dawn" — but A can't know it arrived, so B sends an acknowledgement — but B can't know the ack arrived, so A must ack the ack… forever. No finite number of messages over an unreliable channel can create certainty on both sides. Mapped to messaging: when a consumer processes a message and its ACK is lost, the broker cannot distinguish "processed, ack lost" from "never processed" — it must either redeliver (risking a duplicate) or not (risking a loss). You can't guarantee both no-loss and no-duplicate; you can only pick which risk to take. What systems actually provide is **exactly-once processing (EOS) — achieved by combining at-least-once delivery + idempotency/deduplication + atomic offset-commit-with-output. Kafka offers EOS via idempotent producers (dedupe by producer id + sequence number, preventing duplicate writes on retry) and transactions** (atomically write output messages and commit input offsets, so a read-process-write is all-or-nothing). It works within Kafka; extending it to external side effects (charging a credit card) still needs idempotency on your side. Interview gold: "Exactly-once delivery is impossible; exactly-once processing is achievable with idempotency + atomic commits, and even then only within the transactional boundary."

5.3Ordering#

6Consumer Groups, Rebalancing, Acknowledgement & Redelivery#

Section 5 told you what guarantees the system offers. This section tells you where those guarantees are actually implemented: in the small, unglamorous protocol that decides which consumer process reads which partition, and in the equally small decision of when a consumer records that it is done. Almost every real streaming incident — duplicated emails, silently skipped records, a pipeline that stalls for ninety seconds every two minutes, a consumer group that never stabilises — is a bug in one of those two places. They deserve a section of their own.

6.1The consumer group — what it is and what problem it solves#

Plain definition. A consumer group is a set of consumer processes that share a single group.id string and cooperatively divide a topic's partitions among themselves, such that every partition is assigned to exactly one member of the group at any instant, and the group as a whole collectively reads the entire topic exactly once.

Why it exists. Without groups you would have to solve two problems by hand in every application. First, work division: if you run six copies of your billing service for capacity, something must stop all six from reading the same partition and charging the same card six times. Second, failover: if one of the six dies at 03:00, something must notice and give its partitions to a survivor, without a human. The consumer group is the broker-side machinery that solves both, and it solves them with the same mechanism, because "a member left" and "a member joined" are the same event to the protocol: the assignment must be recomputed.

Where it sits. A group is not a configuration object you create; it springs into existence the first time a consumer sends a request carrying that group.id, and it is stored as in-memory state on one broker plus two durable artefacts in the internal Kafka topic **__consumer_offsets (50 partitions by default, compacted — §9.3): the group's committed offsets** (one record per group.id + topic + partition, whose value is the offset to resume from) and the group's metadata record (its members, its assignment, its generation). Because that topic is replicated like any other, a group survives the failure of the broker that was coordinating it.

Tie-back. Everything in §4.3 — "max useful parallelism equals the partition count" — is a consequence of the exclusivity rule stated above, not an independent fact. One partition, one member: an eleventh consumer against ten partitions is assigned nothing and idles at 0% CPU forever.

6.2The group coordinator — the broker that runs the group#

Plain definition. The group coordinator is a single broker that has been designated as the authority for one specific consumer group: it accepts members' heartbeats, decides when a rebalance is needed, collects the members' subscriptions, distributes the resulting assignment, and stores the group's committed offsets.

Why it exists. Deciding "who owns partition 7 right now" is a consensus-shaped question — if two processes disagree, they both consume partition 7 and duplicate everything. The cheapest correct answer is not to run a consensus protocol per group, but to elect one authoritative decision-maker per group and route every decision through it. Serialising through a single node makes the answer trivially consistent. The durability of that node's decisions is then delegated to Kafka's ordinary replication of __consumer_offsets (File 08), so the coordinator itself can be replaced without losing the group's state.

How the coordinator is chosen — the exact rule. It is not elected. It is computed, deterministically, by every client independently:

coordinator_partition = murmur2(group.id) mod 50        # 50 = offsets.topic.num.partitions
coordinator_broker    = leader of __consumer_offsets-<coordinator_partition>

Worked example: group.id = "billing". Suppose murmur2("billing") = 3,187,442,001, so 3,187,442,001 mod 50 = 1. The coordinator for the billing group is whichever broker currently leads __consumer_offsets-1 — say broker 4. Every billing consumer, on startup, sends a **FindCoordinator** request to any broker, is told "broker 4," and then talks only to broker 4 for group business. If broker 4 dies, ordinary partition leader election (§8.5) moves the leadership of __consumer_offsets-1 to another broker, the clients get NOT_COORDINATOR on their next request, re-issue FindCoordinator, and rediscover the new one. The group's state is not lost because it was never only in broker 4's memory — it was in the replicated log.

Every named thing, individually. offsets.topic.num.partitions (broker config, default 50) is the divisor above; changing it on a live cluster reshuffles every group to a different coordinator and is therefore effectively frozen after first start. offsets.topic.replication.factor (default 3, but famously defaults to 1 on single-broker dev clusters) controls how many copies of the offsets exist — if this is 1 in production, a single broker loss deletes every consumer group's position and every consumer restarts from auto.offset.reset, which is a whole-company incident. group.id is the client-side string; two applications that accidentally share a group.id will steal partitions from each other, each seeing about half the data and each looking, in isolation, like it is working.

What breaks. The classic diagnostic is kafka-consumer-groups.sh --describe --group billing --state, which prints COORDINATOR, ASSIGNMENT-STRATEGY, STATE, and #MEMBERS. A STATE of PreparingRebalance or CompletingRebalance that persists for more than a few seconds means the group is stuck — usually one member that has stopped responding but has not yet timed out, holding the whole group hostage until rebalance.timeout.ms expires.

6.3The rebalance protocol, step by step#

A rebalance is the procedure that recomputes the partition-to-member assignment. It is triggered by exactly four events: (1) a new member sends JoinGroup (scale-up, or a rolling deploy starting a new pod); (2) an existing member leaves gracefully by sending LeaveGroup (a clean shutdown); (3) an existing member is declared dead because it stopped heartbeating for session.timeout.ms or stopped polling for max.poll.interval.ms; (4) the topic's partition count changes, or a member's subscription changes (e.g. a regex subscription now matches a newly-created topic).

Here is the eager (classic) protocol with concrete values. Group billing, topic orders with 6 partitions, three members A, B, C, currently generation = 17, and A holds orders-0, orders-1, B holds 2, 3, C holds 4, 5. A fourth consumer D starts.

 t=0.000  D  --------- JoinGroup(group=billing, member.id="", subscription=[orders]) ----> Coordinator
 t=0.001  Coordinator: group state PreparingRebalance, generation 17 -> 18
          Coordinator returns REBALANCE_IN_PROGRESS to A, B, C on their next heartbeat
 t=0.15   A,B,C: revoke ALL partitions  (<-- THIS is the stop-the-world moment)
          A,B,C: commit offsets for the partitions they are giving up
          A,B,C: --------- JoinGroup(generation=18) ----> Coordinator
 t=0.30   Coordinator has all 4 JoinGroups. It picks the FIRST to arrive (D) as the
          GROUP LEADER -- note: a client, not the broker, computes the assignment.
          Coordinator replies to D with the full member list + their subscriptions.
          Coordinator replies to A,B,C with an empty member list.
 t=0.31   D runs the ASSIGNOR (§6.4) locally:
             A -> orders-0, orders-1     B -> orders-2, orders-3
             C -> orders-4               D -> orders-5
          D: --------- SyncGroup(generation=18, assignment=<the above>) ----> Coordinator
          A,B,C: ------ SyncGroup(generation=18, assignment=<empty>) --------> Coordinator
 t=0.45   Coordinator stores the assignment and answers each member with ITS slice only.
          Group state -> Stable.
 t=0.46   Each member fetches its committed offset for each newly-owned partition and
          resumes. C fetches the committed offset of orders-4; D fetches that of orders-5.
          NOTHING was consumed by anyone between t=0.15 and t=0.46 -- 310 ms of dead time.

Three details in that trace are interview-grade. First, the assignment is computed by a client, not the broker. The coordinator is a mailbox and a referee; the elected group leader (an arbitrary member) runs the assignor. This is deliberate: it lets you plug in a custom assignment strategy by deploying new client code, without touching the cluster. **Second, the generation (also called the epoch) is a monotonically increasing integer that fences stale members.** If A was frozen in a GC pause during the whole rebalance and wakes at t=0.9 still believing it is generation 17 and still owning orders-0, its OffsetCommit and Heartbeat requests are rejected with ILLEGAL_GENERATION, which forces it to rejoin instead of writing offsets for partitions it no longer owns. This is the same fencing token idea File 08 uses to stop a deposed leader from writing. Third, members commit before revoking. If they did not, the offsets processed since the last commit would be lost and the new owner would reprocess them — the rebalance itself would manufacture duplicates.

Every named timeout, individually.

6.4The four assignment strategies — mechanism, what each optimises, and its cost#

The assignor is pluggable via **partition.assignment.strategy, a client config naming one or more assignor classes. All four below solve the same problem — map P partitions across M members — but they optimise different things. The running example throughout: two topics**, orders with 4 partitions and payments with 4 partitions, and three consumers A, B, C, all subscribed to both.

**(a) RangeAssignor — the default in classic Kafka.** Mechanism: for each topic independently, sort that topic's partitions numerically and sort the members lexicographically, then hand out contiguous ranges: the first P mod M members get ⌈P/M⌉ partitions and the rest get ⌊P/M⌋.

orders   (4 parts, 3 members): A -> 0,1    B -> 2      C -> 3
payments (4 parts, 3 members): A -> 0,1    B -> 2      C -> 3
TOTAL:                         A = 4       B = 2       C = 2

What it optimises: co-partitioning. Because the same rule runs per topic and the member ordering is identical, member A gets partition 0 of both topics. If orders and payments are keyed the same way (both by order_id, same partition count), then all facts about one order land on the same consumer, and you can join them locally, in memory, with no network hop. That is the entire basis of how Kafka Streams does joins (§10.5). Cost: systematic imbalance whenever P mod M ≠ 0, and — the killer — the imbalance compounds across topics. Above, A does twice the work of B and C. With ten similarly-shaped topics A would carry ten extra partitions. Use it when you are doing co-partitioned joins and P is an exact multiple of M.

**(b) RoundRobinAssignor.** Mechanism: build one flat list of all topic-partitions across all subscribed topics, sort it, then deal it out one at a time to members in rotation.

Flat list: orders-0, orders-1, orders-2, orders-3, payments-0, payments-1, payments-2, payments-3
Deal:      A         B         C         A         B          C          A          B
TOTAL:     A = orders-0, orders-3, payments-2   (3)
           B = orders-1, payments-0, payments-3 (3)
           C = orders-2, payments-1             (2)

What it optimises: balance — the counts differ by at most one across the whole assignment, not per topic. Cost: co-partitioning is destroyed (orders-0 is on A, payments-0 is on B, so a local join is impossible), and it degrades badly when members have heterogeneous subscriptions — if C subscribes only to payments, round-robin still tries to deal orders partitions into the rotation and skips C, producing lopsided results. Use it when all members subscribe to the same topics and you want even load without local joins.

**(c) StickyAssignor.** Mechanism: a two-goal optimiser. Goal 1 (higher priority): the assignment must be balanced — no member has more than one partition more than any other member that could have taken it. Goal 2: subject to goal 1, preserve as many of the previous assignment's ownerships as possible. It starts from the previous assignment, removes ownerships of partitions whose owner is gone, then moves the minimum number of partitions needed to restore balance.

Worked example: from the round-robin state above, C dies. Round-robin recomputes from scratch and produces A = orders-0, orders-2, payments-1, payments-3 and B = orders-1, orders-3, payments-0, payments-2 — A keeps only 1 of its 3 previous partitions and B keeps only 1 of 3, so six partitions change hands and six sets of consumer-side state (in-memory caches, open DB connections, half-built windows) are discarded. Sticky instead keeps A's three and B's three untouched and merely splits C's two: A = orders-0, orders-3, payments-2, orders-2 and B = orders-1, payments-0, payments-3, payments-1. Two partitions moved instead of six.

What it optimises: the cost of movement. Moving a partition is expensive whenever the consumer holds per-partition state — a Kafka Streams state store must be restored from its changelog (§10.6), a batching consumer loses its half-full batch, a connection pool is rebuilt. Cost: the assignor is more computationally expensive (it is an optimisation pass, not a deal-out loop — noticeable when a group has thousands of partitions), and it still uses the eager revoke-everything protocol, so the stop-the-world pause is not removed, only the restore work afterwards is reduced. Use it when consumers hold state but you are pinned to the eager protocol by an old client.

**(d) CooperativeStickyAssignor — the modern default choice.** Mechanism: identical assignment logic to Sticky, but paired with the incremental cooperative rebalance protocol (§6.5): members do not revoke everything, they revoke only the specific partitions the new assignment takes away from them, and the rebalance runs in two rounds.

What it optimises: the pause, not just the restore. In the C-dies example above, A and B never stop consuming their six retained partitions at all; only the two orphaned partitions are dark, and only for the length of one round trip. Cost: it needs two rebalance rounds instead of one (slightly higher total latency to reach Stable), it requires client version 2.4+ everywhere, and upgrading to it requires a two-step rolling restart — first deploy with partition.assignment.strategy = [CooperativeSticky, RangeAssignor] so mixed old/new members can still agree on the classic protocol, then a second deploy dropping RangeAssignor. Doing it in one step makes the group unable to agree on a common strategy and it will refuse to stabilise.

Decision rule. Use CooperativeSticky unless you have a specific reason not to — it strictly dominates Sticky, and dominates RoundRobin except in the rare case of a very large partition count where the assignor's CPU cost matters. Use Range only when you are deliberately exploiting co-partitioned local joins and you have made P a multiple of M. Never use Range by accident, which is exactly what happens when nobody sets partition.assignment.strategy on an older client.

6.5Stop-the-world versus incremental cooperative rebalancing#

The problem with eager rebalancing. In the classic protocol every member revokes all of its partitions before rejoining, even the ones it is about to be given straight back. This is a stop-the-world pause: the entire consumer group processes zero records from the moment revocation begins until the moment SyncGroup completes. The trace in §6.3 measured that at 310 ms for a trivial group. In a real group it is far worse, because the pause is bounded not by the fast members but by the slowest: the coordinator waits up to rebalance.timeout.ms for every member's JoinGroup, and a member cannot send JoinGroup until it has finished the record batch it is in the middle of. Worked case: a 60-member group whose consumers take up to 20 s to finish a batch, running a rolling deploy of 60 pods one at a time. Each pod restart is two rebalances (one on leave, one on join), each costing up to 20 s of global pause. That is 120 rebalances × ~20 s ≈ 40 minutes during which the group is mostly not consuming, while the producers keep producing. Lag explodes, and if lag crosses retention you lose data (§9.1).

Why incremental cooperative rebalancing was introduced (KIP-429, Kafka 2.4, 2019). The insight is that in the overwhelmingly common cases — one pod added, one pod removed — most members' assignments do not change at all, so revoking them is pure waste. Cooperative rebalancing changes the protocol so that revocation is targeted and deferred:

ROUND 1  Every member sends JoinGroup while STILL CONSUMING its current partitions.
         The leader computes the target assignment and diffs it against the current one.
         It replies with: "keep these; you are LOSING these" -- and crucially it does NOT
         yet hand the lost partitions to their new owner (that would be double ownership).
         Members that lost nothing simply keep going. Members that lost something commit
         offsets for exactly those partitions and revoke exactly those.
ROUND 2  A second, immediate rebalance is triggered. Now the freed partitions are
         unowned, so the leader assigns them to their new owners, who begin consuming.

The consequence: in the "one member of sixty joins" case, fifty-nine members never pause, and the handful of moved partitions are dark for roughly one round trip rather than for the group-wide worst case. The cost is one extra protocol round, and the requirement that all members support the protocol.

**Static group membership (group.instance.id) — the other half of the fix.** Even with cooperative rebalancing, a rolling restart still triggers two rebalances per pod, because the coordinator sees a new member each time (member IDs are generated fresh on every connection). Static membership (KIP-345) lets you set group.instance.id = "billing-7" — a stable identity, typically the Kubernetes StatefulSet ordinal. When a member with a known group.instance.id disconnects, the coordinator does not immediately rebalance; it holds that member's assignment in reserve until session.timeout.ms expires, betting that the same identity will come back. A pod that restarts in 8 seconds with session.timeout.ms = 45000 therefore causes zero rebalances and resumes its exact previous partitions. The trade-off is stated plainly: you have deliberately delayed failure detection. A pod that is genuinely, permanently dead now leaves its partitions unconsumed for the full session.timeout.ms — so static membership converts "frequent short pauses for everyone" into "a rare long pause for one partition set," which is almost always the better trade for a stateful consumer and a bad trade for a latency-critical stateless one.

6.6The failure modes — traced, not named#

**(a) The max.poll.interval.ms eviction, and how it becomes a storm.** This is the single most common Kafka consumer incident. Trace it precisely, with numbers. A consumer has max.poll.records = 500 and max.poll.interval.ms = 300000. Each record requires one call to a downstream API that normally takes 20 ms, so a batch takes 500 × 20 ms = 10 s — comfortably inside the 300 s limit. Now the downstream API degrades and its p99 goes to 900 ms:

How to diagnose it. The log line Member consumer-billing-3 sending LeaveGroup request... due to consumer poll timeout has expired is the definitive signature. The metric pair to watch is JMX kafka.consumer:type=consumer-coordinator-metrics,attribute=rebalance-rate-per-hour (should be ~0 in steady state; anything above a handful per hour is a storm) alongside last-rebalance-seconds-ago. How to fix it, in priority order: (i) reduce max.poll.records so a batch is bounded — going from 500 to 50 turns a 450 s worst case into 45 s, and this is almost always the right first move because it is a one-line change with no downside beyond slightly more poll() overhead; (ii) raise max.poll.interval.ms only if the work genuinely is long, accepting that you have also raised how long a genuinely dead consumer goes undetected; (iii) move slow work off the poll thread onto a worker pool, using pause()/resume() on the partitions so you keep calling poll() (which heartbeats and keeps membership) while returning no new records; (iv) adopt CooperativeSticky so that one member's eviction stops being everyone's problem; (v) fix the actual downstream.

(b) The rebalance storm from flapping infrastructure. A second, distinct cause: session.timeout.ms set too aggressively low (the old 10 s default) against a network with occasional multi-second GC pauses or a noisy Kubernetes node. Members are evicted for being briefly unreachable, rejoin, get evicted again. The tell is that evictions correlate with GC logs or node pressure rather than with batch duration. The fix is to raise session.timeout.ms toward 45–60 s and keep heartbeat.interval.ms at a third of it.

(c) The slow-consumer-holds-the-group-hostage case. During PreparingRebalance, the coordinator waits for every member up to rebalance.timeout.ms. One member stuck in a 4-minute batch means the other fifty-nine wait 4 minutes. This is why rebalance.timeout.ms defaulting to max.poll.interval.ms is a trap: raising the poll interval to "give the slow consumer room" simultaneously raises how long everyone waits for it.

6.7Acknowledgement models — exactly which duplicates or losses each produces#

An acknowledgement is the consumer's statement "I am done with this message; do not give it to me again." In a log-based system it takes the form of an offset commit — writing "group billing has finished orders-3 up to offset 8,142" into __consumer_offsets. In a queue-based system it is a per-message ack/DeleteMessage. The semantics of the whole pipeline hinge on when you do it relative to processing.

**Auto-commit (enable.auto.commit = true, the default).** The client's poll loop commits the offsets of the *last batch returned by poll()* on a timer, auto.commit.interval.ms (default 5,000). Critically, the commit is not triggered by your code finishing — it is triggered by the next call to poll() once 5 s have elapsed. So the commit means "I have handed these records to the application," never "the application succeeded."

Trace the crash. auto.commit.interval.ms = 5000. At t=0 poll() returns offsets 100–199. At t=5.0 the next poll() fires and commits offset 200, then returns records 200–299. At t=6.0 your application is still on record 205 and the process is SIGKILLed by the OOM killer. On restart the consumer resumes at 200. Records 200–205 are reprocessed — a duplicate — which is fine and expected. But now change one thing: suppose your application hands records to a background thread pool and returns immediately. Then at t=5.0 offset 200 is committed while records 100–199 are still queued in the pool. The crash at t=6.0 loses records 150–199 permanently and silently. This is the trap: auto-commit is at-least-once only if your processing is synchronous inside the poll loop; combined with any asynchronous hand-off it silently becomes at-most-once with data loss.

Use auto-commit when: processing is synchronous, fast, and idempotent, and you accept up to auto.commit.interval.ms worth of reprocessing after a crash. Never use it when: processing is asynchronous, or when reprocessing is expensive, or when you need to know exactly what was and was not done.

**Manual commit, after processing (enable.auto.commit = false + commitSync() at end of batch) — at-least-once.** You call poll(), process every record to completion (including any database write or downstream call), and only then commit.

Trace the crash. Records 200–299 processed successfully; the process dies at the instant before commitSync() returns. On restart the consumer resumes at 200 and reprocesses all 100 records. Nothing is lost; up to one full batch is duplicated. This is the correct default for almost every system, and it is the model that makes consumer-side idempotency (§2.5, File 15) non-optional rather than a nicety. Note the subtlety that a commitSync() which times out is ambiguous — the commit may or may not have landed — so the duplicate window is real even when your code sees an exception.

Manual commit, before processing — at-most-once. You call poll(), immediately commit offset 300, then process 200–299.

Trace the crash. The commit lands. The process dies having processed 200–249. On restart the consumer resumes at 300. Records 250–299 are never processed and no error is recorded anywhere — the data is simply gone from the pipeline's perspective while still sitting in the log. Use this only when a lost record is genuinely cheaper than a duplicate one: metrics samples, ephemeral telemetry, a click-heatmap feed. Say the trade-off out loud when you choose it.

**commitSync versus commitAsync.** commitSync() blocks until the coordinator acknowledges and retries internally on retriable errors; it is safe but adds a round trip (a few ms) per batch, which at small batch sizes is a real throughput tax. commitAsync() fires and forgets with a callback; it is faster but does not retry, because a retry could commit an older offset after a newer one has already landed, moving the group backwards and manufacturing duplicates. The idiomatic pattern is therefore commitAsync() in the steady-state loop for speed, plus a final commitSync() in the finally block on shutdown for correctness.

Per-record versus per-batch commit. Committing every record makes the duplicate window one record instead of one batch, but costs one __consumer_offsets write per record — which converts Kafka's beautiful sequential append into per-message bookkeeping and can cut consumer throughput by an order of magnitude. The usual compromise is to commit every N records or every T milliseconds, choosing N so the worst-case replay is tolerable.

Transactional commit (exactly-once processing). When the consumer's output is itself Kafka, you can put the output records and the offset commit into one Kafka transaction (sendOffsetsToTransaction + commitTransaction), so the consumed-offset advance and the produced records land atomically and downstream readers with isolation.level = read_committed never see either half alone. This is the mechanism §5.2 referred to, and its boundary is the Kafka cluster — it cannot make a credit-card charge atomic with an offset commit.

Negative acknowledgement. Queue systems have an explicit "I failed, take it back" signal: RabbitMQ's basic.nack(requeue=true|false) and basic.reject, SQS's ChangeMessageVisibility(handle, 0). Kafka has no nack, because the broker holds no per-message state — the consumer's only options are to not advance the offset (which stalls the partition, §7.3) or to advance past the record and write it somewhere else (a DLQ topic). That asymmetry is a direct consequence of the dumb-broker inversion in §2.12 and is a common source of confusion for engineers moving from RabbitMQ to Kafka.

6.8The queue-world equivalent — visibility timeout and in-flight limits#

The mechanism that plays the role of "membership + poll interval" in a queue is the visibility timeout (SQS) or ack deadline (Google Pub/Sub), taught fully in §2.10. What is worth stating here is the structural correspondence, because interviewers ask you to map one model onto the other:

In-flight limits, concretely. An in-flight message is one that has been delivered but not yet acknowledged and not yet timed out. Every system caps them, and hitting the cap produces a confusing symptom rather than a clean error. SQS caps in-flight at 120,000 messages for a standard queue and 20,000 for a FIFO queue; past that, ReceiveMessage returns empty responses, which looks exactly like an empty queue while ApproximateNumberOfMessagesNotVisible sits pinned at the cap — the diagnosis is always "my consumers are holding messages far longer than the visibility timeout assumes." RabbitMQ caps in-flight per consumer channel with basic.qos(prefetch_count=N) (§2.8); with prefetch_count = 250 and 40 consumers, up to 10,000 messages can be checked out and invisible to the queue-depth metric. Kafka's implicit in-flight bound is max.poll.records times the number of assigned partitions, which is why lowering max.poll.records is simultaneously the fix for poll-interval evictions and the way to bound how much work a single crash replays.

6.9Consumer lag — how to interpret it and how to alert on it#

§2.11 defined lag and its two derived forms (time lag and drain time). This section is about turning it into an alert that is actually actionable, because "alert when lag > 100,000" is the most common and least useful monitor in the industry.

Interpret the shape, not the value. Each of the failure modes in this file writes a distinct signature on the lag graph, and reading the shape tells you which one you have without touching a single log:

   lag                                                    diagnosis
    |    ____                    a spike, then a clean drain to zero
    |   /    \___                -> producer burst; the system worked. NOT an incident.
    |__/         \____
    |
    |        /                   monotonic linear rise, slope = full production rate
    |      /                     -> the consumer group is DEAD or has zero members.
    |____/
    |
    |      /                     monotonic rise at a SHALLOWER slope than production
    |    /                       -> consumers alive but too slow. Scale out (up to the
    |  /                            partition count) or make processing faster.
    |_/
    |
    |    /\  /\  /\              sawtooth: climbs, drops, climbs
    |   /  \/  \/  \             -> REBALANCE STORM (§6.6). Consumption keeps stopping.
    |__/
    |
    |  one partition line rising, all others flat
    |                            -> a poison pill (§7.3), a hot key (skew, §2.7),
    |                               or one wedged consumer instance.

The four alerts worth having. (1) Time lag above the SLA, per consumer group — "the record we are currently processing is more than 120 s older than the newest record in the topic." This is the only alert that maps directly to a promise made to a human, and it is immune to volume: a lag of 5,000,000 on a firehose and a lag of 12 on a trickle can both be 120 s. (2) Drain time negative or infinite — consumption rate ≤ production rate, sustained over a 15-minute window. This fires before the backlog becomes a crisis and is the alert that tells you to scale. (3) Member count is zero, or below expected — because a group that has entirely disappeared often stops reporting lag at all, so a lag-only alert can go quiet exactly when things are worst. (4) Per-partition lag skew — max partition lag divided by median partition lag above, say, 10× — which catches the poison pill and hot-key cases that a summed metric hides completely.

The threshold you must set no matter what. Alert when lag approaches the retention boundary: if retention is 7 days and time lag reaches 5 days, you are 48 hours from deleting unread data. Past that point the consumer's next fetch gets OFFSET_OUT_OF_RANGE, auto.offset.reset silently jumps it to earliest or latest, and the gap is unrecoverable and unlogged (§9.1). This is the mechanism by which "we were a bit behind" becomes "we lost three days of orders."

Tools. kafka-consumer-groups.sh --describe is the manual view. Burrow (LinkedIn) evaluates lag trend over a sliding window of committed offsets and emits a status (OK/WARN/ERR) rather than a number, precisely so you do not have to pick a threshold. Cruise Control (also LinkedIn) goes further and rebalances partitions across brokers to fix the load skew that causes lag in the first place. Confluent Control Center, Datadog's Kafka integration, and kafka_exporter for Prometheus all expose kafka_consumergroup_lag per group per partition — export it per partition, never pre-summed, or alert (4) above is impossible.

Tie-back. Consumer lag is the output signal of every mechanism in this section: assignment decides who can reduce it, rebalancing determines how often reduction stops, and the acknowledgement model determines whether a committed offset means the work is actually done. Read the lag graph and you are reading §6.

7Backpressure, Dead Letters & Poison Pills#

7.1Backpressure#

Sub-Concept: Backpressure. Backpressure is the mechanism by which a slow consumer signals a fast producer to slow down, preventing the consumer (or the broker) from being overwhelmed and running out of memory. In a pull-based system like Kafka, backpressure is natural: consumers pull at their own pace, so a slow consumer simply reads slower and the (durable, disk-backed) log absorbs the backlog — the producer is unaffected and messages wait safely. In push-based systems (some queues, reactive streams), the broker must explicitly throttle or the consumer must advertise capacity (credit-based flow control), or you risk unbounded buffering → OOM (out-of-memory: the process's buffers grow until the machine has no RAM left and the kernel kills it). Kafka's pull model + huge disk-backed retention is why it handles massive producer spikes gracefully: the log is the buffer.

7.2Dead Letter Queue (DLQ)#

Sub-Concept: Dead Letter Queue. A DLQ is a separate queue where messages that repeatedly fail processing are sent after N retries, so they don't block the main queue forever. Instead of retrying a bad message infinitely (or dropping it silently), you move it aside for later inspection/replay. Essential for operability — you get a quarantine of failures to debug, and the pipeline keeps flowing.

7.3Poison Pill#

A poison pill is a message that always fails processing (malformed, references deleted data, triggers a consumer bug). In an at-least-once system, a poison pill is redelivered forever, blocking the partition/queue behind it (head-of-line blocking) and burning resources. Mitigations: retry with a max attempts cap, then route to the DLQ; add jittered backoff between retries; make consumers defensive (validate, catch, don't crash the whole consumer on one bad message).

7.4Retry Strategy#

Robust consumers use exponential backoff + jitter (File 03) on transient failures, a retry cap, and a DLQ for exhausted retries. Distinguish transient failures (network blip → retry) from permanent ones (bad data → DLQ immediately; retrying won't help). Retrying permanent failures just wastes capacity.

8Durability & Replication Inside the Broker — The Write Path#

§4.4 gave you the vocabulary: leader, follower, ISR, acks. This section spends that vocabulary properly. We are going to follow a single record from the moment your application calls producer.send() to the moment the producer's future completes, naming every buffer it sits in, every disk it does or does not touch, and every machine that must agree before anyone calls it "written." Then we will kill a broker at each step in turn and count exactly what is lost.

This matters because "the broker acknowledged it" is not one guarantee, it is five different guarantees depending on three configuration values, and the difference between them is the difference between losing a payment and not.

8.1The write path, traced end to end#

Setup for the whole section: topic orders, partition 3, replication factor 3. Broker 2 is the leader of orders-3; brokers 5 and 7 are followers. The record is a 240-byte JSON order with key user-42.

 APPLICATION                PRODUCER CLIENT (in your JVM)                    NETWORK
 ───────────                ────────────────────────────                    ───────
 send(record) ──▶ [1] serialize key+value to bytes (240 B)
                  [2] partitioner: murmur2("user-42") % 6 = 3
                  [3] metadata cache: leader(orders-3) = broker 2
                  [4] append into the RecordAccumulator:
                        per-partition deque of batches, 32 KB pages
                        batch for orders-3 now holds 512 records / 15.8 KB
                  [5] Sender thread wakes when EITHER
                        batch.size (16384 B) reached  OR  linger.ms elapsed
                  [6] compress the whole batch (compression.type=lz4):
                        15.8 KB ──▶ 4.1 KB on the wire
                  [7] one ProduceRequest per broker, carrying batches for
                      ALL partitions that broker leads  ───────────────────────▶

 BROKER 2 (LEADER of orders-3)
 ────────────────────────────
  [8] network thread reads the request off the socket, hands it to an I/O thread
  [9] validate: CRC of each batch, magic byte, max.message.bytes (1,048,588)
 [10] assign offsets: log currently ends at 1,048,900 (this is the LEO,
      the Log End Offset). The 512 records become 1,048,900 .. 1,049,411.
      LEO advances to 1,049,412.
 [11] write() the compressed batch to the ACTIVE SEGMENT FILE
      (/var/lib/kafka/orders-3/00000000000001048576.log)
      ★ This is a write() into the OS PAGE CACHE. It is RAM. No disk yet. (§0.5)
 [12] update the sparse .index every log.index.interval.bytes (4096) and
      the .timeindex likewise
 [13] the request is parked in the PURGATORY -- a delayed-operation data
      structure holding requests that cannot be answered yet -- IF acks=all

 FOLLOWERS 5 and 7 (they PULL; there is no push replication in Kafka)
 ───────────────────────────────────────────────────────────────────
 [14] each follower runs a ReplicaFetcher thread issuing a Fetch request to
      broker 2 that looks exactly like a consumer's fetch, saying
      "give me orders-3 from offset 1,048,900"
 [15] leader serves those bytes -- usually straight from the page cache it
      just wrote in step 11, so replication normally costs ZERO disk reads
 [16] follower appends to ITS OWN log (again: page cache), advancing its LEO
      to 1,049,412
 [17] follower's NEXT fetch asks for offset 1,049,412 -- and that request is
      itself the acknowledgement. "I am asking for 1,049,412" means
      "I have everything below 1,049,412."

 BACK ON THE LEADER
 ──────────────────
 [18] leader records each follower's LEO. It computes:
        HIGH WATERMARK (HW) = min(LEO of every replica in the ISR)
      Before the fetches: HW = 1,048,900.  After both: HW = 1,049,412.
 [19] HW advancing past 1,049,411 makes the record COMMITTED. Only now is it
      visible to consumers (a consumer can never read beyond the HW).
 [20] purgatory sees the HW moved; it completes the parked ProduceRequest and
      returns {baseOffset: 1048900, timestamp: ...} to the producer.

 PRODUCER
 ────────
 [21] the Future<RecordMetadata> completes; your callback fires.

Two facts in that trace overturn most people's mental model. First, replication is pull-based: followers fetch from the leader using the same protocol consumers use, and a follower's next fetch offset is its acknowledgement. There is no separate replication protocol and no explicit follower ACK message. Second, the acknowledgement does not mean the data is on a disk. Step 11 is a write() into the page cache, which is volatile RAM. Kafka's durability story is not "we fsynced"; it is "N independent machines each have it in their page cache, and N machines do not lose power simultaneously." That is a deliberate design choice, and §8.3 explains its exact cost.

Every producer-side name, individually.

Every broker-side name, individually.

8.2acks — the three values, each with its exact failure trace#

**acks is a producer-side setting that names the moment the broker is allowed to answer.** It does not change what the broker does with the data; it changes when you find out.

**acks=0 — the producer never waits.** The producer writes the request to the socket and completes the future immediately. It does not wait for a response at all.

Exact loss trace: the producer sends 512 records. The TCP connection to broker 2 was reset half a millisecond earlier; the write goes into a socket buffer that is then discarded. The producer's callback fires with success. There is no exception, no metric, no log line. 512 orders vanished. Because there is no response, acks=0 also disables retries entirely (there is nothing to retry on) and defeats the idempotent producer (which needs the broker's sequence-number response). Throughput gain: real but smaller than people assume — perhaps 10–25% over acks=1 — because the producer was pipelining up to 5 in-flight requests anyway. Use only for: high-volume telemetry where a lost sample is genuinely worthless, e.g. per-request latency histograms sampled at 1%. Never for anything a human or a ledger will notice.

**acks=1 — the leader answers as soon as it has appended to its own page cache.** Step 20 of the trace happens right after step 12; followers are not waited for.

Exact loss trace, step by step:

The honest summary: acks=1 means "durable against a process crash, not against a machine loss." A Kafka broker restarting cleanly does not lose page-cache data (the OS flushes on clean shutdown); a kernel panic, power loss, or instance termination does. Use it when: the data is reconstructible from an upstream source, or the loss window (single-digit milliseconds of records on one partition) is genuinely acceptable — log shipping and metrics pipelines usually qualify.

**acks=all (a.k.a. acks=-1) — the leader answers only when the high watermark has advanced past the record.** Because HW = min(LEO over the ISR), this means every replica currently in the ISR has the record in its page cache.

Latency cost, with numbers: you have added one follower fetch round trip. Within one datacenter that is typically 1–3 ms added to p50 and perhaps 10–20 ms to p99 (a GC pause on a follower delays the HW). Across availability zones add the inter-AZ RTT, ~1–2 ms in AWS. Across regions it becomes 60–150 ms, which is why cross-region acks=all is almost always the wrong architecture — you mirror asynchronously instead (§14.1).

What it does NOT guarantee: that the data is on any disk (it is in three page caches), and — crucially — that there was more than one replica in the ISR at the time. That second gap is the subject of the next subsection and is the single most commonly misunderstood point in all of Kafka.

8.3min.insync.replicas — the setting that makes acks=all mean something#

Plain definition. min.insync.replicas is a broker- or topic-level setting that specifies the minimum number of replicas that must currently be in the ISR for a write with acks=all to be permitted at all. If the ISR has fewer members than this, the leader rejects the produce request with NotEnoughReplicasException (or NotEnoughReplicasAfterAppendException) rather than accepting it.

Why it exists — the worked case that proves the point. acks=all says "wait for all in-sync replicas." Note the words carefully: all in-sync replicas, not all replicas. Now consider a topic with replication.factor = 3, acks=all, and the default min.insync.replicas = 1:

The system silently degraded from "three copies" to "one copy" without changing a single configuration value and without any producer seeing an error. **This is why acks=all alone is not a durability guarantee — it is a durability guarantee only in combination with min.insync.replicas ≥ 2.**

The correct configuration and its arithmetic. The standard production triple is:

replication.factor    = 3      (topic)
min.insync.replicas   = 2      (topic or broker)
acks                  = all    (producer)

Read it as: "a write must be on at least 2 machines before it is acknowledged, and there are 3 machines so we can lose 1 and still accept writes." The general formula for how many broker failures you can tolerate while still serving writes is replication.factor − min.insync.replicas. With RF=3/minISR=2 that is 1. With RF=3/minISR=3 it is 0 — a single broker restart for a routine kernel patch takes the topic read-only, which is why minISR=RF is a configuration people set once, get paged for, and never set again. With RF=5/minISR=3 it is 2, which is what you choose when you need to survive an AZ failure plus a concurrent single-node failure.

What happens when the floor is breached. With minISR=2 and the ISR down to {2}, the leader returns NotEnoughReplicasException to producers. The producer treats this as retriable and keeps retrying until delivery.timeout.ms (120 s) expires, after which your application gets an exception and must decide what to do — buffer, spill to disk, shed load, or fail the user request. **This is the trade stated plainly: min.insync.replicas = 2 converts a silent durability loss into a loud availability loss. That is the CAP-shaped choice of File 19 made concrete at the level of one config line: Kafka with minISR ≥ 2 is a CP** system for that partition — when it cannot guarantee the write is safely replicated, it refuses the write.

A subtlety worth knowing. min.insync.replicas is only consulted for acks=all producers. A producer using acks=1 against the same topic writes happily while the ISR is at 1. So the setting is a contract between the topic and well-behaved producers, not an absolute broker enforcement — which is why mature organisations pin acks=all in a shared client library rather than trusting each team's config.

8.4ISR membership — how a replica falls out, catches up, and rejoins#

Plain definition. The ISR (In-Sync Replica set) is the leader-maintained set of replicas that are considered sufficiently caught up to be eligible to become leader without data loss. It always contains the leader itself.

Why it exists. A fixed "wait for all replicas" rule would mean the slowest machine in the cluster dictates every write's latency, and one sick disk would halt production cluster-wide. A fixed "wait for a majority" rule (what Raft does, File 18) would mean tolerating F failures requires 2F+1 machines. The ISR is a third design: a dynamic, leader-adjusted membership set that lets slow replicas be ejected rather than waited for, so RF=3 can tolerate 2 failures for durability purposes while a majority protocol with 3 nodes tolerates only 1. The price is that the ISR must itself be stored somewhere authoritative — which is what the controller quorum is for.

The exact rule for falling out. The only criterion in modern Kafka is **replica.lag.time.max.ms** (default 30,000, lowered from 10,000 in older versions): if a follower has not sent a fetch request that caught it up to the leader's LEO within this window, it is removed from the ISR. Note what this rule is not: it is not a message-count threshold. Kafka used to have replica.lag.max.messages, and it was removed in 0.9 for a specific and instructive reason — a threshold like "1,000 messages behind" is meaningless against a variable rate. During a producer burst of 50,000 records/s, a perfectly healthy follower that is 200 ms behind is 10,000 messages behind and gets ejected; during idle periods a genuinely broken follower stays in the ISR because no messages are arriving for it to be behind on. Time-based lag is rate-independent, and that is the whole point.

The full lifecycle, with values. Leader broker 2, followers 5 and 7. replica.lag.time.max.ms = 30000.

What breaks. (a) ISR flapping — a replica that oscillates in and out because it is chronically at ~30 s of lag. Each shrink/expand is a controller write and a metadata propagation to every broker and client, so a cluster with thousands of partitions flapping generates a metadata storm. Diagnose with IsrShrinksPerSec/IsrExpandsPerSec — in a healthy cluster these are zero, and any sustained nonzero value is a sick broker, a saturated network link, or a disk about to fail. (b) Under-replicated partitions — the metric UnderReplicatedPartitions (count of partitions where ISR size < RF) is the single best cluster-health number to alert on; nonzero for more than a few minutes means you are one failure away from a durability event. (c) **UnderMinIsrPartitionCount** — partitions where the ISR has dropped below min.insync.replicas, i.e. partitions that are currently **rejecting acks=all writes**. This should page immediately. (d) A slow follower during a rolling restart is normal and expected: each broker restart makes its replicas fall out and rejoin, which is why you wait for UnderReplicatedPartitions to return to 0 before restarting the next broker. Skipping that wait is the classic way to turn a routine upgrade into an outage.

8.5Leader election, the leader epoch, and unclean leader election#

Clean leader election. When a leader dies, the controller (the elected brokerology-wide coordinator: a ZooKeeper-elected broker in old Kafka, an active member of the Raft-based KRaft quorum in modern Kafka) picks a new leader for each affected partition from the surviving members of that partition's ISR, preferring the first entry of the replica assignment list (the "preferred leader") when it is in the ISR. Because every ISR member by definition has every record up to the HW, no committed record is lost. Records that were above the HW — appended to the old leader but not yet replicated — are discarded, but no consumer ever saw them and, with acks=all + minISR ≥ 2, no producer was ever told they succeeded.

The leader epoch — why truncation is safe. Each partition carries a leader epoch: an integer incremented every time leadership changes, persisted in the leader-epoch-checkpoint file next to the log segments (§2.7). When the old leader returns, it does not simply keep its extra records. It issues an OffsetsForLeaderEpoch request to the new leader asking "what was the end offset of epoch 12?", learns that epoch 12 ended at 1,048,900, and truncates its own log back to 1,048,900 before starting to fetch as a follower. Without epochs (Kafka before 0.11) truncation was done by comparing high watermarks, which had known divergence bugs where two replicas could end up with different bytes at the same offset. The leader epoch is the same fencing-token idea as the rebalance generation in §6.3 and as the term number in Raft (File 18).

Unclean leader election — the precise data loss it permits. Set unclean.leader.election.enable = true (topic or broker level; **default false in modern Kafka**) and you authorise the controller, when no ISR member survives, to elect a leader from the out-of-sync replicas instead of leaving the partition offline.

Exact trace:

The trade stated plainly. Unclean leader election is a lever between availability and durability with no middle setting. false says: "I would rather this topic be completely unavailable than serve a log that has silently lost records." true says: "I would rather keep flowing and lose some data than stop." For orders, payments, ledgers, and anything a human will reconcile — **false, always** (and this has been the default since Kafka 0.11 precisely because too many people were losing data without knowing). For a metrics or clickstream topic where availability is the product and a gap is invisible noise — true is defensible, and you should say so explicitly rather than leaving it to a default. The metric to watch is OfflinePartitionsCount: nonzero means partitions with no leader at all, which under unclean=false is exactly the situation this setting created deliberately.

8.6Why Kafka's ISR is NOT consensus — and how it relates to Files 08 and 18#

This is a distinction interviewers use to separate people who have read about Kafka from people who understand it, so be precise.

What consensus (File 18: Raft, Paxos) does. A consensus protocol lets a set of nodes agree on a value with no external authority, tolerating up to F failures with 2F+1 nodes, by requiring a quorum (majority) to accept each decision. Its correctness comes from the overlap property: any two majorities of the same set share at least one node, so a new leader's majority necessarily contains a node that saw the previous leader's last committed decision. Nothing outside the group is needed.

What Kafka's ISR replication does. A single leader accepts writes, followers pull, and a record is committed when every member of a dynamically-adjusted set has it. Crucially, **the ISR itself is not agreed by the replicas — it is decided by the leader and *stored in an external strongly-consistent store***: ZooKeeper historically, and the KRaft controller quorum today. So the durability of the ISR-membership decision is delegated to a system that does run consensus.

The three concrete differences.

Where consensus is nevertheless present in Kafka. Modern Kafka runs KRaft, a genuine Raft implementation, for the metadata layer: which brokers exist, which topics exist, who leads each partition, and what each partition's ISR is. Those decisions are replicated to a controller quorum (typically 3 or 5 controllers) by Raft, with all of Raft's guarantees. So the accurate sentence is: "Kafka uses consensus for metadata and ISR-based primary–backup replication for data." That split is a deliberate engineering choice — running Raft per partition (as Pulsar's BookKeeper and Redpanda do differently, §14.1) means a quorum round trip per write per partition, which for a cluster with 200,000 partitions is a very different cost profile than one metadata quorum plus cheap pull replication.

Tie-back to the rest of the bible. File 08 covers leader–follower replication, replication lag, and the general durability-versus-latency curve — Kafka's ISR is one specific, unusually well-documented point on that curve. File 18 covers Raft and Paxos in full, including the majority-overlap argument sketched above and the leader-epoch/term fencing idea that §8.5 uses. File 19 (CAP/PACELC) is what you are actually configuring when you choose min.insync.replicas: 2 is the C-over-A choice, 1 is the A-over-C choice, and the "else latency" half of PACELC is exactly the acks=1 versus acks=all decision in §8.2.

9Retention, Compaction & Storage Economics#

A queue's storage question is trivial: a message occupies space until it is acknowledged, then it does not. A log's storage question is the hard one, because the log keeps data whether or not anyone has read it, and therefore someone must decide when to throw it away. That decision — retention — is simultaneously a durability policy, a replay-window policy, a compliance policy, and the largest line item on your cluster's bill. This section covers the three mechanisms Kafka offers (delete-by-time/size, compact-by-key, and tier-to-object-storage), the arithmetic that turns a throughput number into a disk purchase order, and the guidance for choosing.

9.1Retention by time and by size — the two delete policies#

Plain definition. Retention is the rule that decides which records the broker is allowed to delete. Under cleanup.policy = delete (the default), a record is eligible for deletion when it is older than retention.ms or when the partition's total size exceeds retention.bytes — whichever triggers first.

Why it exists. Disks are finite (File 04's whole premise) and an append-only log grows without bound. Unlike a queue, the log cannot use "everyone has read it" as its deletion criterion, because the log does not know who "everyone" is — the entire point of §2.9's design is that adding a new consumer group tomorrow costs the broker nothing, which is only true because the broker keeps no per-consumer state about what has been read. So deletion must be driven by a policy the operator sets, not by consumption.

Every named setting, individually.

The capacity arithmetic, worked with real numbers. This is the calculation you should be able to do on a whiteboard in ninety seconds. The formula is:

raw disk = ingest_rate_bytes_per_sec × retention_seconds × replication_factor
                                                    ÷ compression_ratio
           ─────────────────────────────────────────────────────────────────
           and then ÷ target_disk_utilisation (never plan for 100% full)

Worked example — a clickstream topic. 400,000 events/second, average serialized event 800 bytes, retention 7 days, RF=3, lz4 compression achieving 4:1 on JSON, and you plan to run disks at 60% so that a broker failure's re-replication has somewhere to land.

ingest              = 400,000 × 800 B          = 320,000,000 B/s = 320 MB/s
after compression   = 320 MB/s ÷ 4             =  80 MB/s   (this is what hits disk)
per day             =  80 MB/s × 86,400        =   6.91 TB/day
7 days, one replica =   6.91 × 7               =  48.4 TB
× RF 3              =  48.4 × 3                = 145.2 TB  of raw stored bytes
÷ 60% utilisation   = 145.2 ÷ 0.6              = 242 TB of provisioned disk

Sanity-check the network too, because it usually binds before disk:
  inbound to leaders            =  80 MB/s
  replication traffic           =  80 × (3-1) = 160 MB/s   (leaders push to 2 followers)
  consumers, 4 groups           =  80 × 4     = 320 MB/s
  total cluster                 = 560 MB/s = 4.5 Gbit/s
  on 6 brokers with 10 GbE      = 0.75 Gbit/s each -- fine
  on 6 brokers with 1 GbE       = 0.75 Gbit/s each -- SATURATED. The NIC is the wall.

Two lessons fall out of that arithmetic and both are interview-grade. First, replication multiplies everything — disk, and network twice over (once to write, once per follower). Going from RF=2 to RF=3 is a 50% increase in your entire storage bill, which is why "just set RF=5" is not free. Second, consumer fan-out multiplies read bandwidth linearly while costing zero storage — the exact inverse of the SNS/subscription model in §2.9. A tenth consumer group adds 80 MB/s of reads and 0 bytes of disk.

What breaks. (a) Retention shorter than consumer lag. This is the failure §2.11 and §6.9 both warned about, and here is its precise mechanism: the retention thread deletes segment 00000000000001048576.log; a lagging consumer's next fetch asks for offset 1,050,000, which no longer exists; the broker returns OFFSET_OUT_OF_RANGE; the client consults **auto.offset.reset** — earliest (jump to the oldest surviving record, skipping the deleted gap), latest (jump to the tail, skipping everything accumulated), or none (throw NoOffsetForPartitionException and stop). The default in most clients is latest, which means the pipeline silently skips forward and the gap is discovered days later by someone reconciling totals. **Set auto.offset.reset = none on any pipeline where a gap is unacceptable, and handle the exception loudly. (b) retention.bytes set per topic in someone's head but per partition in the config — the 50× surprise above. (c) A skewed key filling one partition** (§2.7) hits that partition's retention.bytes first, so that one partition has a much shorter effective time retention than its siblings — which produces the baffling symptom "we have 7-day retention but this one key's history only goes back 9 hours."

9.2Segments and the active segment — why deletion is coarse-grained#

Plain definition. A partition's log is not one file; it is a sequence of segment files, each named by the base offset of its first record (§2.7 showed the directory listing). Exactly one segment — the last — is the active segment, and it is the only one being appended to. All the others are closed and immutable.

Why this structure exists. Three reasons, each of which would be sufficient on its own. (1) Deletion. You cannot cheaply remove bytes from the front of a single large file — the filesystem has no "truncate from the head" operation, so you would have to rewrite the whole file. You can unlink() a whole file in constant time. Segments make expiry a file deletion instead of a data copy. (2) Bounded index size. Each segment has its own .index and .timeindex, which are memory-mapped; keeping segments at 1 GB keeps each index small enough to mmap comfortably. (3) Bounded recovery. After an unclean shutdown the broker must verify CRCs and rebuild indexes for the segments that might be corrupt — only the active segment can be torn, so recovery scans one segment per partition, not the whole log.

The mechanism, step by step, with values. log.segment.bytes = 1073741824 (1 GB), retention.ms = 604800000 (7 days), ingest 80 MB/s compressed across 20 partitions = 4 MB/s per partition.

What breaks. (a) Huge segments on a low-volume topic. A topic ingesting 100 KB/day with the default 1 GB segment size and 7-day time-based roll will hold everything for a week before rolling and then another week before expiring — so its effective retention is roughly double what you configured. Lower log.segment.bytes (e.g. 100 MB) or log.roll.ms on such topics. (b) Tiny segments on a high-volume topic. Setting log.segment.bytes = 10485760 (10 MB) on a topic doing 4 MB/s per partition rolls a segment every 2.5 seconds, producing 34,000 segments per partition per day, each with 2 index files — you will exhaust the process's file-descriptor limit (ulimit -n, and nofile in the systemd unit) and the broker will die with Too many open files. That is one of the most common Kafka production failures and its root cause is almost always segment sizing or partition count. (c) A compliance requirement to delete one user's records ("right to erasure"). Segment granularity means you cannot delete one record. The real answers are either key-based compaction with a tombstone (§9.3) or encrypting per-user data with a per-user key and destroying the key — crypto-shredding — which turns "delete the bytes" into "delete the 32-byte key" and is the standard pattern for GDPR compliance on an immutable log.

9.3Log compaction — in full#

Plain definition. Log compaction (cleanup.policy = compact) is a retention policy that, instead of deleting records by age, retains at least the most recent record for every distinct key and garbage-collects the older records that share that key. The result is a log whose size is bounded by the number of distinct keys, not by the number of events.

Why it exists. §4.6 gave the headline: it turns a topic into a durable table. Here is the underlying need. Many streams are not really event streams — they are state streams: user-42 → {name, email, tier}, updated whenever the profile changes. If such a topic used time retention, then after 7 days the record for a user who has not changed their profile in a month would be deleted, and a new consumer replaying from the beginning would never learn that user exists. The topic would no longer be a complete description of the state. Compaction fixes exactly this: it guarantees that replaying a compacted topic from offset 0 yields the current value of every key that has ever existed and has not been explicitly deleted. That property — and only that property — is what makes a Kafka topic usable as a materialized-state source, a changelog, or a database replacement for lookup data.

Where it sits. Compaction is performed by a pool of background threads called the log cleaner (log.cleaner.threads, default 1), which operates on closed segments only — never the active segment — and is enabled cluster-wide by log.cleaner.enable (default true since 0.9). A partition's log is conceptually split into two regions:

 ┌──────────────────────── CLEANED (the "tail") ────────────┬──── DIRTY (the "head") ────┬─ ACTIVE ─┐
 │  compaction has run here; at most one record per key      │ not yet compacted; may     │ being    │
 │  offsets are NON-CONTIGUOUS (gaps where records were      │ contain many records for   │ appended │
 │  removed) -- this is legal and consumers handle it        │ the same key               │          │
 └───────────────────────────────────────────────────────────┴────────────────────────────┴──────────┘
                                                             ^
                                                        the "cleaner point"

The cleaner's mechanism, step by step, with a worked example. Topic user-profiles, cleanup.policy = compact.

Log before cleaning (offset: key → value):

 off 0 : user-1  -> {tier: free}
 off 1 : user-2  -> {tier: free}
 off 2 : user-1  -> {tier: pro}
 off 3 : user-3  -> {tier: free}
 off 4 : user-2  -> {tier: pro}
 off 5 : user-1  -> {tier: enterprise}
 off 6 : user-3  -> null            <-- a TOMBSTONE
 ─────── end of dirty region; offsets 7+ are the active segment ───────

Log after cleaning (note the offset gaps — offsets are not renumbered, ever):

 off 4 : user-2  -> {tier: pro}
 off 5 : user-1  -> {tier: enterprise}
 off 6 : user-3  -> null

Offsets 0, 1, 2, 3 simply do not exist any more. A consumer that asks for offset 2 gets offset 4 — the next existing record. Any code that assumes consecutive offsets is broken on a compacted topic, and that is by design.

Tombstones and the delete-retention window — the subtle part. A tombstone is a record with a key and a null value. It means "this key is deleted." The cleaner cannot simply drop a tombstone the first time it sees it, because a slow consumer that is currently at offset 0 has not yet seen the deletion; if the tombstone vanished, that consumer would replay user-3 → {tier: free} at offset 3 and never see the delete, ending with a phantom user in its materialized view forever.

So tombstones get a grace period: **delete.retention.ms (default 86,400,000 = 24 hours). The rule is precise: a tombstone survives compaction until at least delete.retention.ms after the cleaner first passes over it**; only on a later pass, once that window has elapsed, is the tombstone itself removed. The operational contract this creates is: *any consumer that could ever need to observe deletes must complete a full read of the log within delete.retention.ms.* If your bootstrap replay takes 30 hours and delete.retention.ms is 24 hours, your rebuilt state can contain records that were deleted. Raise delete.retention.ms above your worst-case full-replay time.

Every other compaction setting, individually.

How compaction differs from retention — the comparison.

Where compaction is used in Kafka itself. __consumer_offsets is compacted, keyed by (group, topic, partition) — which is exactly why a consumer group's position survives forever without the topic growing without bound (§6.1). Kafka Streams' state-store changelogs are compacted (§10.6). Debezium's CDC topics are typically compacted so the topic converges to "current row per primary key" (§11.3).

What breaks. (a) The cleaner falling behind. Watch JMX kafka.log:type=LogCleanerManager,name=max-dirty-percent and time-since-last-run-ms. If a single cleaner thread cannot keep up with the dirty rate — common on high-cardinality topics — the log grows unboundedly despite compact, and bootstrap replays get slower and slower. Fix by raising log.cleaner.threads and log.cleaner.dedupe.buffer.size. (b) A cleaner thread dying (historically on OutOfMemoryError from an undersized dedupe buffer, or on a corrupt record) is silent: compaction simply stops for the whole broker, and the only symptom is disk usage that no longer plateaus. Alert on time-since-last-run-ms. (c) Assuming compaction is immediate. It is not: it is a background pass gated by dirty ratio and segment closure, so writing a tombstone does not remove the old value "now," and the active segment is never compacted — meaning the most recent writes always contain duplicates per key. Any consumer of a compacted topic must therefore still handle seeing several values for one key and take the last.

9.4Tiered storage — moving the log to object storage#

Plain definition. Tiered storage (Kafka KIP-405, generally available in Kafka 3.6+; earlier and more mature in Confluent Platform, Apache Pulsar, and Redpanda) splits a partition's log into a local tier on the broker's own disk and a remote tier in an object store (S3, GCS, Azure Blob — File 09), and moves closed segments from local to remote once they age past a threshold.

Why it exists — the economics that force it. Take the 145 TB from §9.1's worked example. On AWS gp3 EBS at roughly $0.08/GB-month, 145 TB costs about $11,600/month in storage alone, and — the part people forget — that storage is welded to compute: to hold 145 TB across brokers with a sane per-broker disk of 6 TB you need at least 24 brokers, whether or not your throughput needs 24 brokers. You are buying CPU and RAM to hold bytes nobody is reading. Meanwhile S3 Standard is about $0.023/GB-month, and critically you store only one copy in S3 (S3 does its own replication internally and charges you for one), so the same 7 days of data is 48.4 TB × $0.023 = ~$1,110/month — roughly a 10× reduction — and the cluster can shrink to however many brokers your throughput actually needs, maybe 6. That gap is why tiered storage exists.

What moves and what stays, precisely.

The latency consequence — say this number. A consumer reading at the tail hits the page cache and is served with sendfile at effectively RAM speed: sub-millisecond, no disk I/O at all (§4.5). A consumer reading local historical data hits the disk: single-digit milliseconds. A consumer reading remote data triggers an S3 GET of a segment range: 50–200 ms of first-byte latency, after which throughput is fine (S3 will happily stream at hundreds of MB/s, and brokers prefetch). So the shape of the trade is: **tiered storage makes historical reads ~100× slower to start but leaves tail reads completely untouched**, which is exactly the right trade because 95%+ of all Kafka reads are tail reads. The workloads that hurt are bootstrapping a brand-new consumer group over months of history, and any disaster-recovery replay — both are throughput-bound rather than latency-bound, so they mostly survive it.

Why it changes retention economics fundamentally. Before tiered storage, "how long do we keep it?" was a question with a cluster-shaped answer: doubling retention meant doubling brokers. Teams therefore kept 3–7 days and built a separate, entirely different pipeline (Kafka → S3 via Kafka Connect → Parquet → Spark) to serve historical needs, and then had to reconcile two representations of the same data with different schemas, different tooling, and different bugs. After tiered storage, retention is a line item you can dial to months or years without touching the cluster shape, and the same consumer API, the same offsets, and the same ordering guarantees apply to a record from ten minutes ago and one from ten months ago. That collapse of two pipelines into one is the actual point; the cost saving is the headline but the operational simplification is the deeper win.

Its costs, stated plainly. (a) Request charges. S3 charges per GET (~$0.0004 per 1,000). A badly-behaved consumer replaying with tiny fetches can generate millions of GETs; brokers mitigate by fetching whole segments and caching them locally, but you must watch it. (b) Egress. Reading from an object store in a different region or out to the internet incurs data-transfer charges that can dwarf the storage saving. (c) A new failure domain. Your log's availability now depends on the object store's; an S3 regional event makes historical reads fail while tail reads keep working, a partial-failure mode that did not exist before. (d) Operational immaturity relative to the core log — the remote-storage manager plugin is comparatively new in Apache Kafka, and recovery paths (a broker that dies mid-upload, orphaned remote segments after a topic delete) are the rough edges. (e) No compacted topics.

Product note. Pulsar's tiered storage has existed since 2018 and is architecturally more natural there because BookKeeper already separates storage from serving (§14.1). Redpanda and WarpStream go further: WarpStream runs stateless brokers with S3 as the only tier, eliminating local disks and inter-AZ replication charges entirely, at the cost of several hundred milliseconds of produce latency — a completely different point on the latency/cost curve, and the right one for high-volume analytics ingestion.

9.5Choosing retention — the practical guidance#

Retention is not one number; it is the answer to four separate questions, and you should take the maximum of the four.

The decision rules.

Tie-back. Retention is the parameter that turns the log from §3.2's abstract "replayable stream" into a concrete, budgeted system. Every replay capability this file has celebrated — reprocessing after a bug (§13), onboarding a new consumer group for free (§2.9), rebuilding a state store (§10.6), bootstrapping a CDC consumer (§11.3) — is only true inside the retention window, and this section is where you decide how wide that window is and what it costs.

10Stream Processing on Top of the Log#

Everything so far treats the consumer as a black box that "processes" a record. This section opens that box. Stream processing is the discipline of computing continuous, incrementally-updated results — counts, sums, joins, aggregations, alerts — over an unbounded sequence of records, and it turns out to require a specific and non-obvious set of machinery: a notion of which time you mean, a way to decide when a result is final, a place to keep partial results, and a way to make all of it survive a crash.

Why it belongs in this file. A log is a substrate, not an application. The reason Kafka is interesting is that once your data is an ordered, replayable, partitioned log, you can build derived streams and derived tables on top of it, and — the key insight — the same machinery that gives the log its guarantees gives the processor its guarantees. Exactly-once processing (§5.2) is only achievable because offsets and outputs live in the same transactional system. State restoration after failure is only cheap because state can be written to a compacted topic (§9.3). Parallelism in the processor is the partition parallelism of §2.7 with a different name. Stream processing is not a separate topic bolted on; it is what the log was for.

10.1Stateless versus stateful operators#

An operator is one transformation step in a processing topology — a node in a directed graph where records flow along the edges.

Stateless operators compute their output for a record using only that record. map (transform each record), filter (drop records failing a predicate), flatMap (one record in, zero-or-many out), branch/split (route to different output streams by predicate), foreach (side effect, no output), and merge (interleave two streams) are the whole family. Their defining property, and the reason they are boring in the best sense: an instance can be killed and restarted anywhere, at any time, with no recovery step at all — it just resumes reading from its committed offset. There is nothing to restore because there was nothing kept. Scaling is trivial and rebalancing (§6.5) costs a poll cycle. If your entire pipeline is stateless, you do not need a stream-processing framework; a plain consumer loop is genuinely sufficient, and saying so in an interview is a mark of judgement.

Stateful operators must remember something across records. count, sum, min/max, aggregate, reduce, all windowed operations, all joins, and deduplication are stateful. Consider count by user_id: to emit "user-42 has now clicked 17 times," the operator must have the number 16 in front of it. That number is state, and its existence changes everything about the operator's lifecycle:

Tie-back. The stateless/stateful split is the single best predictor of how hard a streaming system will be to operate. Every operational difficulty in the rest of this section — restore time, state size, rebalance cost, checkpointing — exists only because of state.

10.2Event time versus processing time versus ingestion time#

The three timestamps, defined.

Why the distinction exists, and why it is not pedantry. In a batch world these three collapse: you process yesterday's file today and every record in it is "yesterday." In a streaming world they diverge constantly and by large amounts, because of four unavoidable realities: mobile devices go offline and buffer events for hours; producers retry across network partitions; partitions are consumed at different rates so records interleave out of order; and a consumer that fell behind and is catching up processes an hour of history in a minute. If you compute "sales in the 14:00 hour" using processing time, you have not computed sales in the 14:00 hour — you have computed "records our consumer happened to handle between 14:00 and 15:00," which is an operational artefact of your own infrastructure, not a fact about the business.

A concrete example showing them diverge. A rider opens a ride-hailing app on a train, enters a tunnel, and the app buffers events. Kafka topic ride-events, one partition for clarity.

Now compute the same business question three ways: "how many ride requests happened in the 14:05–14:06 minute?"

When each is nonetheless the right choice. Processing time is correct when the question is genuinely about your infrastructure ("how many records did we handle per second?", "alert if we processed zero records in the last minute") and it is the cheapest option — no buffering, no watermarks, results emitted immediately. Ingestion time is a pragmatic middle: monotonic, requires no trust in producer clocks, and is fine when producer-to-broker latency is small and bounded — a good default for server-side events where the producer is your own service in the same datacenter. Event time is required whenever the result is a statement about the world, whenever results must be reproducible on replay, and whenever producers can be delayed. It is also the most expensive, because it forces the machinery of §10.3.

What breaks. Trusting CreateTime means trusting every producer's clock. A mobile fleet includes devices with clocks years wrong, and a device claiming event time of 2035 will, in an event-time windowing job, advance the watermark to 2035 and cause every subsequent legitimate record to be classified as late and dropped. The standard defence is to clamp: reject or re-stamp any event time more than N minutes ahead of broker time, and treat implausibly old timestamps as a separate quarantined stream.

10.3Watermarks — bounding lateness#

Plain definition. A watermark is a marker flowing through the stream that asserts: "I do not expect to see any more records with event time earlier than T." It is a heuristic claim about completeness, not a guarantee.

Why it exists. Event-time windowing poses an unanswerable question: when can I emit the result for the 14:05 window? Never emitting is useless. Emitting immediately is wrong, because record B from §10.2 arrives fourteen minutes later. There is no way, in a distributed system with buffering devices, to know that no more 14:05 records will come — a phone could be off for a month. So the system must make a bounded guess, and the watermark is that guess made explicit and made a first-class data item so it can propagate through the whole topology.

Where it sits. In Flink, a watermark is a real record interleaved into the stream alongside data records, generated by a WatermarkStrategy at the source and propagated through every operator; an operator with multiple inputs takes the minimum of its inputs' watermarks (because it is only as complete as its least-complete input, which also means one idle partition holds back the whole job's watermark — a famous operational gotcha, solved by marking idle sources with withIdleness(Duration)). In Kafka Streams there is no separate watermark object; the equivalent is stream time, the maximum event timestamp observed so far per task, combined with a per-window grace period. The concept is identical; the plumbing differs.

The mechanism, step by step, with values. The most common generator is bounded-out-of-orderness with a configured maximum delay, say 2 minutes: watermark = (max event time seen so far) − 2 minutes.

 records arriving (event times)      max seen      watermark = max − 2min
 ────────────────────────────────    ────────      ──────────────────────
 14:03:11                            14:03:11      14:01:11
 14:03:50                            14:03:50      14:01:50
 14:05:40                            14:05:40      14:03:40
 14:04:12  <- out of order, fine     14:05:40      14:03:40   (does not go backwards)
 14:07:20                            14:07:20      14:05:20
 14:08:05                            14:08:05      14:06:05  ◀── crosses 14:06:00, so the
                                                                 [14:05,14:06) window FIRES
 14:05:55  <- LATE. Its window already fired.

Two properties are essential. A watermark never moves backwards — it is monotonic by construction (it is a running max minus a constant), because an operator that un-fired a window would be a nightmare. And watermark progress is driven by data, not by wall clock: if the stream goes silent, the watermark freezes and windows never fire, which is why a low-traffic topic can leave results unemitted for hours. Flink's withIdleness and Kafka Streams' wall-clock punctuator exist to break that deadlock.

Every named setting, individually.

What happens to late data — the three options, with their consequences.

The trade-off, stated plainly. Watermark delay is a dial between latency, completeness, and cost. Short delay: results fast, more records late, more corrections or more loss. Long delay: results complete, but every result is published late by that amount and every open window's state is held in memory for that much longer. There is no setting that gives you all three, and being able to say that sentence — and then ask "what is the business tolerance for a wrong-but-fast number?" — is the senior answer to any streaming design question.

10.4Windowing — tumbling, hopping, and session, compared over one stream#

Plain definition. A window is a finite, bounded subset of an unbounded stream, defined by time (or count), over which an aggregate can actually be computed. You cannot "count all events" on an infinite stream — the answer never exists — so you count events per window.

The single input stream used for every example below. One key (user-42), event times only, over a ten-minute span:

 event times:  10:00:30   10:01:10   10:02:45   10:06:20   10:06:50   10:07:10   10:09:40
 label:           e1         e2         e3         e4         e5         e6         e7

(a) Tumbling windows. Mechanism: fixed size, non-overlapping, contiguous. Window boundaries are computed from an epoch, not from the first record, so TumblingWindows.of(5 min) produces [10:00,10:05), [10:05,10:10), and so on, identically on every instance and on every replay. Every record belongs to exactly one window.

 time  10:00      10:01      10:02      10:03      10:04      10:05      10:06      10:07      10:08      10:09      10:10
       |----------|----------|----------|----------|----------|----------|----------|----------|----------|----------|
 e:    e1         e2              e3                                          e4 e5      e6                    e7
 W1:   [====================== 10:00 – 10:05 ======================]                            count = 3  (e1,e2,e3)
 W2:                                                              [====================== 10:05 – 10:10 ====] count = 4  (e4,e5,e6,e7)

Pros: the cheapest option — state is O(1) window per key at a time, each record touches one window, and results partition the timeline with no double counting, so summing all windows equals the total. Cons: boundary blindness. A burst spanning 10:04:50–10:05:10 is split into two windows of 1 and 1, and an alert rule of "3 events in 5 minutes" never fires even though 2 events happened in 20 seconds. Complexity: one aggregate per key per window. Use when: you are producing periodic reports — hourly revenue, daily active users, per-minute request counts for a dashboard.

(b) Hopping windows (Kafka Streams' name) / Sliding windows (Flink's name for the same thing). Mechanism: fixed size, advanced by a smaller hop (or slide), so windows overlap. size = 5 min, advance = 1 min produces [10:00,10:05), [10:01,10:06), [10:02,10:07)… **Every record belongs to size ÷ advance windows simultaneously — here, 5 of them.**

 time  10:00  10:01  10:02  10:03  10:04  10:05  10:06  10:07  10:08  10:09  10:10
 e:    e1     e2         e3                          e4 e5   e6                e7
       [------- 10:00–10:05 -------]                                    count 3  (e1,e2,e3)
              [------- 10:01–10:06 -------]                             count 2  (e2,e3)
                     [------- 10:02–10:07 -------]                      count 3  (e3,e4,e5)
                            [------- 10:03–10:08 -------]               count 3  (e4,e5,e6)
                                   [------- 10:04–10:09 -------]        count 3  (e4,e5,e6)
                                          [------- 10:05–10:10 -------] count 4  (e4,e5,e6,e7)

Pros: it is the right shape for rolling-window alerting — "more than 3 failed logins in any 5-minute period" is a question about any 5-minute period, not about the aligned ones, and only overlapping windows can answer it. Note that in the trace above, tumbling reported a maximum of 4 while hopping surfaces the same 4 but also shows the sustained level across the burst. Cons: the cost multiplies. Each record is written into size ÷ advance windows, so state, CPU, and output volume are all 5× the tumbling equivalent here, and with size = 1 hour, advance = 1 second it is 3,600× — a configuration that looks innocent and destroys a cluster. Downstream also sees every result 5 times with different boundaries, so the sink must handle that. Complexity: O(size ÷ advance) per record. Use when: alerting, rate detection, or any "in any N-minute period" rule — and always compute size ÷ advance before shipping it.

Terminology warning, because it is a real interview trap. Kafka Streams calls the overlapping fixed-size window a hopping window, and reserves "sliding window" for a different construct (SlidingWindows.ofTimeDifferenceWithNoGrace) where windows are defined by record proximity — a window exists around each record covering records within the time difference, so windows are created only where data exists. Flink and most of the literature call the overlapping fixed-size window a sliding window and have no separate "hopping." When someone says "sliding window," ask which they mean.

(c) Session windows. Mechanism: windows have no fixed size. A window is defined by a gap (an inactivity timeout): records for a key are collected into the same session as long as consecutive records are less than gap apart; a gap larger than the timeout closes the session and starts a new one. With gap = 2 minutes:

 time  10:00  10:01  10:02  10:03  10:04  10:05  10:06  10:07  10:08  10:09  10:10
 e:    e1     e2         e3                          e4 e5   e6                e7
       [=== SESSION A ======]                        [== SESSION B ==]      [SESSION C]
        e1,e2,e3                                      e4,e5,e6                e7
        10:00:30 – 10:02:45                           10:06:20 – 10:07:10     10:09:40
        duration 2m15s, count 3                       duration 50s, count 3   duration 0, count 1

  gaps:  e3 →  e4  =  3m35s  > 2m  ⇒ close A, open B
         e6 →  e7  =  2m30s  > 2m  ⇒ close B, open C

Mechanism detail that matters: sessions merge. If a late record arrives at 10:04:00, it is within 2 minutes of both e3 (10:02:45) and e4 (10:06:20)? No — but a record at 10:05:00 would be within gap of e4 and within gap of nothing before it, whereas a record at 10:04:30 bridges e3 and e4 and causes sessions A and B to be merged into one session, with their aggregates combined and the two old session records tombstoned. This merge-on-arrival behaviour is why session windows are the most expensive to implement and why their output is inherently a changelog (sessions appear, grow, and get replaced) rather than a sequence of final values.

Pros: it is the only window shape that matches human behaviour, which is bursty and has no relationship to clock boundaries. "How long was this user's visit?", "how many pages per browsing session?", "group these IoT readings into machine-run cycles" are all natively session questions and are badly served by any fixed window. Cons: unbounded window length (a user who clicks every 119 seconds for eight hours produces one eight-hour session whose state you must hold), the merge complexity above, results that arrive at unpredictable times (a session's result cannot be emitted until gap has elapsed with no activity plus the grace period), and per-key state that is far larger than fixed windows. Complexity: O(open sessions per key), with merges. Use when: the natural unit of analysis is a burst of activity by one entity.

The comparison table.

Decision rule. Ask what the question is. If it is "per hour/day/minute," tumbling — and resist any temptation to overlap, because you will pay for it. If it contains the words "in any" or "rolling," hopping — and immediately compute size ÷ advance and check that the multiplier is affordable (keep it ≤ 10 in practice). If the entity's activity is bursty and the boundaries should be discovered rather than imposed, session. And if none of them fit — if you need "the last 100 events regardless of time" — you want a count window or a plain stateful operator, not a time window at all.

10.5Joins and co-partitioning#

A join combines records from two streams (or a stream and a table) that share a key. It is the most state-hungry operation in streaming and the one with the strictest structural requirement.

The three shapes. A stream–stream join requires a window, because two unbounded streams have no point at which a match is impossible — clicks JOIN impressions WITHIN 10 minutes ON ad_id means each side buffers 10 minutes of records in state so the other side can find them. A stream–table join is an enrichment: each arriving stream record looks up the current value of the key in a materialized table (orders JOIN customers ON customer_id to attach the customer's tier), needs no window, and is cheap — the table is just a local key-value store. A table–table join maintains a continuously-updated joined table, re-emitting whenever either side changes.

Co-partitioning — the hard requirement. For a join to be computable locally, the matching records must be on the same instance. That requires all of: (1) both topics keyed by the join key, (2) the same number of partitions in both topics, and (3) the same partitioning function. If orders has 12 partitions and customers has 8, order_id=X lands on orders-5 and its customer lands on customers-3, which is served by a different instance, and Kafka Streams will refuse to start with a TopologyException about co-partitioning rather than silently produce wrong results. **This is why §6.4's RangeAssignor exists** — it deliberately gives partition n of every subscribed topic to the same member, so co-partitioned topics are co-located. The escape hatch when co-partitioning is impossible is a global table (GlobalKTable in Kafka Streams): every instance holds a full copy of the (small) lookup table, so no partitioning constraint applies, at the cost of every instance storing the whole thing — appropriate for country codes, currency rates, or a feature-flag table, not for a customer table with 200 million rows.

10.6State stores and their changelog topics#

Plain definition. A state store is the local, embedded key-value store where a stateful operator keeps its working data — the running counts, the open windows, the join buffers.

Where it sits, concretely. In Kafka Streams the default implementation is RocksDB, an embedded log-structured-merge-tree key-value store (File 11) running inside your application's process, writing to local disk with an in-memory block cache. It is not a database you connect to; it is a library and a directory (state.dir, default /tmp/kafka-streams — a default that has caused real data loss on machines that clear /tmp on boot, so always override it). Flink's equivalents are the HashMapStateBackend (JVM heap, fastest, bounded by heap) and the EmbeddedRocksDBStateBackend (on disk, supports state far larger than memory, and the only one supporting incremental checkpoints).

Why local state exists at all — the "why" that matters. The obvious alternative is to keep aggregate state in an external database: on each record, GET the current count, increment, PUT it back. Do the arithmetic on that. At 200,000 records/second, that is 400,000 network round trips per second to your database, each costing perhaps 1 ms of latency and consuming a connection — you have made the stream processor's throughput a function of a remote database's capacity, and you have reintroduced exactly the synchronous coupling (§0.2) that the log existed to remove. Local state replaces a network round trip with a memory or local-disk access — roughly 100 ns or 100 µs instead of 1 ms, a 10–10,000× improvement, and it scales with the processor rather than against a shared bottleneck. That is the entire argument for embedding the store.

The problem local state creates, and the changelog that solves it. Local state is on the local disk of an instance that will die. When it does, the count for every key it owned is gone, and the only recovery would be to replay the entire input topic from the beginning — hours or days.

So every state store is backed by a changelog topic: an internal Kafka topic, named <application.id>-<store-name>-changelog, with the same partition count as the store's input, to which every single write to the store is also produced. Recovery is then: start a new instance, read the changelog partition from offset 0, apply every record to a fresh local store, and you have reconstructed the exact state. The state lives locally for speed and durably in Kafka for recovery — and the durability is the broker's replication (§8), which you already trust.

Why the changelog is a compacted topic — this is the tie-back to §9.3, and it is not incidental. A changelog is a stream of key → new value writes. Under time retention, a key written once and never updated would be deleted after 7 days, and a restore would silently omit it — the restored state would be wrong, and wrong in a way no error would reveal. Under compaction, the topic retains the latest value for every key forever, so a restore is guaranteed complete, and the topic's size is bounded by the number of distinct keys rather than the number of updates. A counter updated a billion times for 5 million users is 5 million records after compaction, not a billion. Compaction is precisely the property that makes "a Kafka topic" a viable durable backing store for a table, which is why §9.3 called it the thing that turns a log into a table. Kafka Streams creates these topics with cleanup.policy = compact automatically; windowed stores use compact,delete with retention set to the window size plus grace, because old windows genuinely should disappear.

Standby replicas. num.standby.replicas (default 0) tells Kafka Streams to have other instances also consume the changelog and maintain a warm shadow copy of a store they do not currently own. When the owner dies, an instance with a warm standby takes over in seconds instead of restoring for twenty minutes. The cost is a full extra copy of the state per standby (disk on every standby instance) plus the changelog read bandwidth. **For any job with more than a few gigabytes of state, num.standby.replicas = 1 is not optional — it is the difference between a 30-second failover and a 30-minute outage.**

Interactive queries. Because the state store is a real local database holding a materialized view, Kafka Streams lets you query it directly over HTTP from within your application (store.get(key)), with a metadata API that tells you which instance owns a given key so you can proxy the request. This is a genuinely powerful pattern — your streaming job is your read API, with no separate serving database — and its limits are equally real: no secondary indexes, no ad-hoc queries, and availability tied to the streaming job's rebalances.

10.7Checkpointing and how exactly-once actually works in a stream processor#

The problem. A processor's correctness state is spread across three places: the input offsets (how far it has read), the local state (its accumulated aggregates), and the output records (what it has already emitted). A crash must not leave these inconsistent. If offsets advance but state is lost, records are skipped. If state persists but offsets do not, records are double-counted. If output was emitted and then the whole thing is replayed, downstream sees duplicates.

Flink's answer: distributed snapshots via the Chandy–Lamport algorithm. The JobManager periodically injects a checkpoint barrier — a special marker carrying a checkpoint ID — into every source. The barrier flows downstream with the records.

  source ──r r r ▮ r r r──▶  operator A ──r r ▮ r r──▶  operator B ──▶ sink
                 ▲                          ▲
                 barrier n                  barrier n

  When an operator receives barrier n on ALL of its inputs (it BLOCKS the inputs
  that arrived early -- "barrier alignment" -- until the slowest arrives), it:
     1. snapshots its own state to durable storage (S3/HDFS)
     2. forwards barrier n downstream
     3. resumes processing
  When every operator has acknowledged barrier n, checkpoint n is COMPLETE and the
  JobManager records it. Source offsets are part of the snapshot.

The insight is that a barrier cleanly separates "records before the checkpoint" from "records after," so the collection of per-operator snapshots taken at barrier n forms a globally consistent cut of the whole pipeline — even though no two operators paused at the same wall-clock instant. On failure, the whole job restarts from checkpoint n: sources rewind to the offsets in the snapshot, operators reload their state, and everything after the cut is recomputed. execution.checkpointing.interval (commonly 10 s–1 min) controls frequency; shorter means less recomputation after a failure but more snapshot overhead. Unaligned checkpoints (a Flink option) skip the blocking alignment by snapshotting the in-flight buffers too — better under backpressure, at the cost of larger snapshots.

But that only makes state and offsets consistent — the sink is the hard part. Recomputing after a rewind re-emits output. Making that exactly once end to end requires the sink to be one of:

**Kafka Streams' answer: processing.guarantee = exactly_once_v2. Rather than global snapshots, Kafka Streams leans entirely on Kafka transactions. For each commit interval, one task opens a Kafka transaction and atomically writes: (a) its output records to the output topics, (b) its state-store changelog records, and (c) its input offset commits** to __consumer_offsets (via sendOffsetsToTransaction). All three land or none do. Downstream consumers set isolation.level = read_committed, which makes the broker withhold records from aborted or open transactions, so they never observe a half-completed step. commit.interval.ms defaults to 100 ms under exactly-once (versus 30,000 ms otherwise), because the transaction boundary is also the latency of making results visible — exactly-once in Kafka Streams therefore costs you visible latency, not just throughput, and that is the trade to state.

The boundary, restated because it is the most-missed point in the file. Both mechanisms give exactly-once processing with respect to systems that participate in the protocol — Kafka topics, offsets, and checkpointed state. Neither can make an external side effect exactly-once. If your operator calls a payment API, a replay calls it again, and no barrier or transaction prevents it. The answer there is unchanged from §5.2: an idempotency key and a dedupe table on the receiving side (File 15). exactly_once_v2 is a guarantee about a closed world, and you must know where its walls are.

10.8The operational realities#

State size is the number that governs everything. Work it out before you build. A count by user_id over 80 million monthly users with a 16-byte key and an 8-byte long is ~2 GB of raw entries, but RocksDB overhead, SST-file amplification, and the block cache realistically make it 4–8 GB on disk per replica of that state — times the number of standby replicas, times every instance if you were careless enough to use a global table. A windowed aggregate multiplies that by the number of retained windows: a 1-hour hopping window advancing every minute retains 60 windows per key, so the same 80 million users becomes ~120 GB. That arithmetic is why the first design question for any streaming job is "what is the cardinality of the grouping key, and how many windows do I retain?"

Restore time after failure is the number that governs your SLA. Restoring a state store means reading its changelog partition end to end and applying every record to RocksDB. At a realistic restore rate of 30–60 MB/s per task (limited by RocksDB write throughput, not the network), a 40 GB state store takes 11–22 minutes during which that partition produces no output. Mitigations, in order of effectiveness: (1) standby replicas (§10.6) reduce it to seconds; (2) persistent volumes — if the instance restarts on the same node with its state directory intact, Kafka Streams reads the store's local checkpoint file, restores only the delta since, and is back in seconds, which is the entire reason to run streaming jobs as Kubernetes StatefulSets with PVCs rather than Deployments; (3) more partitions so each task's state is smaller and restores in parallel; (4) aggressive changelog compaction (min.cleanable.dirty.ratio lowered, §9.3) so there is less to replay.

Rebalancing stateful tasks is where §6 and §10 collide. Every point in §6.5 about incremental cooperative rebalancing is multiplied in severity here, because moving a partition now means moving gigabytes of state, not just a cursor. Kafka Streams therefore ships its own assignor, the **StreamsPartitionAssignor, which is sticky by design and adds two things the generic assignors lack: it weighs standby-replica placement (never put a standby on the same instance as the active), and it performs warmup-based probing rebalances** — when a task should move to a new instance, that instance first catches up a copy of the state in the background while the old owner keeps serving, and only when the copy is within acceptable.recovery.lag (default 10,000 records) does a follow-up rebalance actually hand over. probing.rebalance.interval.ms (default 10 minutes) controls how often it re-checks. The effect is that scaling a stateful streaming job out is not instant — you add an instance and it becomes useful minutes later, once it has warmed up. Plan capacity ahead of demand accordingly; a stateful streaming job cannot autoscale on a one-minute trigger the way a stateless service can.

The other realities in brief but explained. Uneven state distribution: a hot key gives one task 10× the state of its peers, and since state does not split below a partition, the only fix is to change the key or increase partitions — and changing the partition count of a stateful job invalidates all existing state, because the key→partition mapping changed (§2.7), forcing a full reprocess. Schema evolution: the serialized bytes in a state store were written by the old version of your code; a deploy that changes the value class must either be backward-compatible or reset the application (kafka-streams-application-reset.sh) and reprocess. Disk on the streaming nodes: people size streaming pods for CPU and are astonished when RocksDB fills the ephemeral disk; state stores need real, monitored, provisioned volumes. Monitoring: the metrics that matter are consumer lag (§6.9) for input health, task-process-latency and RocksDB's compaction/write-stall metrics for processing health, and the count of dropped-late records (§10.3) for correctness health.

Tie-back. Stream processing is the payoff for everything earlier in this file: partitions give it parallelism (§2.7), consumer groups give it fault-tolerant work assignment (§6), the broker's replication gives its checkpoints and changelogs durability (§8), and compaction gives its state stores an unbounded-in-time but bounded-in-size home (§9.3). Take any one of those away and the stream processor cannot be built.

11Integration Patterns — Outbox, CDC, Saga, Claim-Check#

The previous sections were about the broker. This one is about the seam between the broker and everything else — specifically the database. Almost every real service both stores state and publishes events, and the moment it does both, a family of problems appears that no amount of broker tuning can solve, because the problem is not inside the broker at all: it is that a database transaction and a broker publish are two separate operations with no shared commit.

This section covers the four canonical patterns that address that seam, plus the composition shapes you will draw on a whiteboard. They appear in interviews constantly, and the dual-write problem in §11.1 is the "why" behind all of them.

11.1The dual-write problem — stated precisely and traced both ways#

Plain definition. A dual write is any operation where a single logical change must be recorded in two independent systems — canonically, "save the order to PostgreSQL" and "publish OrderCreated to Kafka" — with no transaction spanning both. Because the two systems have separate commit protocols, there is no instant at which both are guaranteed to have happened, and a crash in the gap leaves them inconsistent.

Why it exists — the reason you cannot just fix it. The obvious fix is a distributed transaction: two-phase commit (2PC), in which a coordinator asks every participant "can you commit?" (the prepare phase), and only if all say yes does it tell them "commit" (the commit phase). The XA standard implements this and PostgreSQL supports it (PREPARE TRANSACTION). It is not used here for three concrete reasons. First, Kafka does not support XA at all — its transactions are internal to Kafka and cannot enlist an external resource manager, so the option is simply unavailable. Second, 2PC is blocking: if the coordinator crashes after the prepare phase, every participant sits holding locks, unable to decide, until the coordinator returns — a prepared transaction in PostgreSQL holds its locks indefinitely and will eventually stall the whole database. Third, it couples availability multiplicatively: the transaction can only commit if every participant is up, so a system of five 99.9%-available services has a 99.5%-available transaction. 2PC trades exactly the decoupling (§0.4) that you adopted messaging to obtain, which makes it self-defeating here.

So the dual write cannot be made atomic. It must instead be made unnecessary — that is what the outbox and CDC patterns do.

Failure order A — database first, then publish.

 1. BEGIN
 2. INSERT INTO orders (id, user_id, total, status) VALUES ('ord_881', 42, 99.00, 'PLACED')
 3. COMMIT                                    ◀── the order now exists, durably
 4. producer.send("orders", OrderCreated{ord_881})
 5.        ✗ CRASH between 3 and 4 (process killed, pod evicted, broker unreachable,
             delivery.timeout.ms expires, or the process succeeds at 4 but the network
             ate the request and the retry budget ran out)

Result: the order exists in the database but no event was ever published. The warehouse never picks it, the confirmation email never sends, the analytics count is short by one, and the search index never learns about it. Nothing anywhere is in an error state — the API returned 201, the database looks perfect, and the inconsistency is invisible until a human reconciles counts weeks later. This is a lost event, and it is the more common of the two orders because "commit, then publish" is what a naive implementation writes.

Failure order B — publish first, then database.

 1. producer.send("orders", OrderCreated{ord_881}).get()   ◀── acknowledged by the broker
 2. BEGIN
 3. INSERT INTO orders ...
 4.        ✗ CRASH, or the INSERT violates a unique constraint, or the tx deadlocks
             and is rolled back, or the database fails over mid-commit

Result: an event exists for an order that does not and never will exist. Downstream consumers act on a phantom: the warehouse reserves stock for it, the email service tells the customer their order is confirmed, the ledger records revenue. The event cannot be un-published (Kafka is append-only; you can only publish a compensating event, which every downstream consumer must be written to understand). This is an invented event, and it is worse than a lost one because the damage propagates outward and the correction requires cooperation from every consumer.

The naive "fixes" and why each fails.

The property we actually need, and which the next two patterns provide: the decision to publish must be made durable by the same commit that makes the state change durable. Once both facts are in one transaction, the publish becomes a retryable, at-least-once delivery of an already-durable intent, and at-least-once we know how to handle (idempotency, File 15).

11.2The transactional outbox#

Plain definition. The transactional outbox pattern writes the event to an ordinary table in the same database and the same transaction as the business state change, and then a separate process — the message relay — reads that table and publishes to the broker, deleting or marking rows once published.

Why it works. Both writes are now in one local ACID transaction, so they are atomic by the database's own guarantee — no distributed protocol needed. Either the order and its outbox row both exist, or neither does. The dual write has been eliminated, not coordinated. What remains is a single-system publish problem (relay → broker), which is retryable because the intent is durably recorded.

The schema, concretely.

CREATE TABLE outbox (
    id             UUID        PRIMARY KEY,          -- also used as the Kafka message key / dedupe key
    aggregate_type TEXT        NOT NULL,             -- 'Order'      -> becomes the topic
    aggregate_id   TEXT        NOT NULL,             -- 'ord_881'    -> becomes the partition key
    event_type     TEXT        NOT NULL,             -- 'OrderCreated'
    payload        JSONB       NOT NULL,             -- the event body
    created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
    published_at   TIMESTAMPTZ NULL                  -- NULL = not yet published
);
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;

The partial index matters: without it, the relay's WHERE published_at IS NULL scan degrades as the table grows, and a full scan every 200 ms against a large table is a self-inflicted database outage.

The write side, step by step.

BEGIN;
  INSERT INTO orders (id, user_id, total, status)
       VALUES ('ord_881', 42, 99.00, 'PLACED');
  INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
       VALUES ('7c2f…', 'Order', 'ord_881', 'OrderCreated',
               '{"order_id":"ord_881","user_id":42,"total":99.00}');
COMMIT;                      -- ◀── ONE commit. Both rows, or neither.

The relay side — the two implementations.

(a) Polling publisher. A background loop, every 200 ms:

SELECT id, aggregate_type, aggregate_id, payload
  FROM outbox
 WHERE published_at IS NULL
 ORDER BY created_at
 LIMIT 500
   FOR UPDATE SKIP LOCKED;      -- lets N relay instances run concurrently without
                                -- fighting over the same rows; SKIP LOCKED makes each
                                -- instance take a different batch instead of blocking

For each row: producer.send(topic=aggregate_type, key=aggregate_id, value=payload), wait for the ack, then UPDATE outbox SET published_at = now() WHERE id = .... Trace the crash: the relay publishes row 7c2f successfully and dies before the UPDATE. On restart it sees published_at IS NULL, publishes it again, and the consumer receives the event twice. That is at-least-once, by construction and unavoidably — the relay has the exact same "did my ack get lost?" ambiguity as §5.2's Two Generals. Pros: trivially simple, works on any database, easy to reason about and to test. Cons: polling latency (you add half the poll interval to every event's latency — 100 ms average at a 200 ms interval), continuous query load on the operational database even when idle, and the table must be pruned or it grows forever (DELETE FROM outbox WHERE published_at < now() - interval '7 days', ideally on a partitioned table so pruning is a DROP PARTITION rather than a mass delete).

(b) Log-tailing relay. Instead of querying the table, the relay reads the database's replication log and reacts to inserts into the outbox table — i.e. it is CDC (§11.3) pointed at the outbox. Debezium ships an Outbox Event Router transform that does exactly this, mapping aggregate_type to the destination topic and aggregate_id to the key. Pros: near-zero added latency (single-digit milliseconds), zero additional query load on the database, and ordering is exactly the commit order. Cons: requires the CDC infrastructure of §11.3 and its operational burden. Decision rule: start with polling if you have no CDC; use log-tailing the moment you have Debezium running for anything else, because it is strictly better once the fixed cost is paid.

Ordering. Because aggregate_id is the Kafka key, all events for one order land on one partition and are strictly ordered (§5.3). Events for different aggregates have no ordering guarantee, which is correct and desirable. If you need global ordering the outbox does not help you — you need one partition, with all the costs §5.3 described.

The consumer-side obligation — this is not optional. At-least-once delivery means the consumer must be idempotent, and "must" here is not a style preference: without it the outbox pattern converts "occasionally lose an order" into "occasionally send two confirmation emails and charge twice." The mechanism is the one File 15 develops in full: the event carries a stable, producer-assigned unique ID (the outbox row's id), and the consumer maintains a processed-events table keyed by that ID, inserting it in the same transaction as its own side effect:

BEGIN;
  INSERT INTO processed_events (event_id) VALUES ('7c2f…');   -- PK violation if duplicate
  INSERT INTO shipments (order_id, ...) VALUES ('ord_881', ...);
COMMIT;
-- On unique-violation of processed_events: ROLLBACK and ACK the message. Already done.

Note the shape: the consumer solves its own dual-write with the same trick. The dedupe record and the side effect are in one transaction, so "did I do this?" and "I did this" cannot disagree. When the side effect is external (a payment API), the equivalent is passing the event ID as the API's idempotency key — again, File 15.

What breaks. (a) The relay falls behind during a broker outage; the outbox table grows to millions of rows and the polling query slows down, which slows the relay further. Alert on SELECT count(*) FROM outbox WHERE published_at IS NULL and on the age of the oldest unpublished row — that age is your event-publication lag, the outbox equivalent of consumer lag (§6.9). (b) **Two relay instances without SKIP LOCKED either deadlock or double-publish everything. (c) Someone writes to orders without writing to outbox — a migration script, an admin console, a different service sharing the database. The outbox is only as complete as the discipline around it, which is the strongest argument for CDC, where completeness is enforced by the database itself. (d) Payload bloat**: putting a 5 MB document in the outbox makes the operational database do the work of a message broker; use claim-check (§11.4).

11.3Change Data Capture (CDC)#

Plain definition. Change Data Capture is the practice of observing a database's own replication log — the ordered, durable record the database already writes of every committed change — and turning each entry into an event on a stream. PostgreSQL calls its log the WAL (Write-Ahead Log); MySQL calls it the binlog; Oracle calls it redo logs; MongoDB exposes it as the oplog via change streams; SQL Server has a native CDC feature.

Why it exists, and why it is the deepest idea here. Every ACID database already maintains a totally-ordered, durable, exactly-committed log of every change — it must, because that log is how it implements crash recovery and how it feeds its own read replicas (File 08). CDC's insight is that this log is already the event stream you were trying to construct by hand, and it is authoritative in a way an application-level publish can never be: a change is in the WAL if and only if it committed. No code path can bypass it — not a migration, not an admin tool, not a stored procedure, not a different service writing to the same table. CDC makes the dual-write problem structurally impossible rather than merely well-managed, because there is only one write, and the event is derived from it.

Where it sits. Between the database and Kafka, as a connector process. The canonical implementation is Debezium, an open-source set of Kafka Connect source connectors (Postgres, MySQL, MongoDB, SQL Server, Oracle, Db2, Cassandra) originally built at Red Hat. Kafka Connect itself is Kafka's framework for running connectors as a distributed, fault-tolerant worker cluster with its own offset storage.

How it works, step by step, with the real mechanics. For PostgreSQL:

Why CDC is lower-impact than polling — quantified. The alternative is a polling query: SELECT * FROM orders WHERE updated_at > :last_seen. Compare them honestly.

That last row is the one people miss and it is a genuine correctness bug, not a performance concern.

What CDC costs — stated plainly.

Canonical uses. Keeping a search index (File 13) in sync with the system of record; invalidating or filling a cache (File 02) on write; feeding a data warehouse without nightly batch ETL; the strangler-fig migration, where a new service consumes CDC from the legacy database's tables and builds its own state while the monolith keeps writing, until the day you flip the write path; and replicating between heterogeneous databases.

11.4Claim-check#

Plain definition. The claim-check pattern stores a large payload in an object store (File 09 — S3, GCS, Azure Blob) and puts only a reference to it — the "claim check," typically a URI plus metadata and a content hash — into the message. The consumer receives the small message and fetches the large body itself.

Why it exists. Brokers are optimised for many small messages and are actively bad at large ones. The hard limits make the point concretely: Kafka's max.message.bytes defaults to 1,048,588 bytes (~1 MB) and exceeding it fails the produce with RecordTooLargeException (§2.7); SQS caps a message at 256 KB; Google Pub/Sub caps at 10 MB; RabbitMQ has no hard cap but degrades badly because messages are held in memory. And you can raise Kafka's limit — you can set max.message.bytes = 52428800 (50 MB) — but consider what that does: every replica stores 50 MB (×3 with RF=3 = 150 MB of cluster disk per message), every follower fetch moves 50 MB, replica.fetch.max.bytes and the consumers' max.partition.fetch.bytes must all be raised in lockstep or replication silently stalls, one such message monopolises a partition's throughput and blows past max.poll.interval.ms while the consumer downloads it (§6.6), and page-cache locality — the reason Kafka is fast at all (§4.5) — is destroyed because a handful of messages evict everything else. Raising the limit does not solve the problem; it relocates the cost onto the most expensive storage tier you own.

Object storage, by contrast, is designed for exactly this: unbounded object size, ~$0.023/GB-month (versus replicated broker disk at 3× a much higher per-GB price), and parallel range-GETs.

The mechanism, step by step.

 PRODUCER                                        BROKER            CONSUMER
 ────────                                        ──────            ────────
 1. have a 40 MB scanned invoice PDF
 2. compute sha256 = "e3b0c442…"                                    
 3. PUT s3://invoices-raw/2026/07/ord_881/e3b0c442.pdf
       (idempotent: the key is content-addressed, so a retry
        overwrites with identical bytes -- no duplicate object)
 4. publish a ~400-byte message ─────────────▶  orders topic
       {
         "order_id":  "ord_881",
         "claim":     "s3://invoices-raw/2026/07/ord_881/e3b0c442.pdf",
         "bytes":     41_943_040,
         "sha256":    "e3b0c442…",
         "content_type": "application/pdf",
         "expires_at": "2026-08-20T00:00:00Z"
       }
                                                        ─────────▶ 5. receive message
                                                                    6. GET the object
                                                                    7. verify sha256
                                                                    8. process

Why the order matters absolutely: store first, then publish. If you publish first, a consumer can receive the claim check and issue the GET before the upload finishes, getting a 404 for an object that will exist in two seconds. Consumers then need retry logic that cannot distinguish "not yet" from "never." Uploading first makes the message's existence imply the object's existence, which is a much stronger and simpler contract. Note that this is itself a dual write (§11.1) — S3 write plus broker publish — but a benign one: the failure mode is an orphaned object (wasted money) rather than a lost or invented event, which is why the ordering choice is exactly the opposite of the one you would make for a database.

Every design element, individually. The content hash (sha256) does three jobs: it makes the storage key content-addressed so uploads are idempotent under retry, it lets the consumer verify integrity, and it enables deduplication of identical payloads. **bytes lets the consumer decide whether it can handle the object before downloading (a 4 GB claim check should route somewhere else). expires_at publishes the lifecycle contract explicitly so a consumer that is 30 days behind knows to fail loudly rather than get a mystery 404. Presigned URLs** are the right mechanism when the consumer is in a different trust domain — a time-limited signed URL grants read access without giving the consumer IAM credentials to the bucket, and its expiry must exceed your worst-case consumer lag.

Lifecycle and cleanup — the part everyone forgets. The message expires per the topic's retention (§9.1); the object does not expire at all unless you make it. Three concerns:

Trade-offs. Gains: the broker stays fast and small, storage cost drops by an order of magnitude, and the payload size ceiling effectively disappears. Costs: two round trips instead of one (added latency: an S3 GET's first byte is 50–200 ms); a second failure domain (the object store can be unavailable while the broker is fine); the orphan-and-lifecycle management above; and a loss of atomicity — the message and its body are no longer one unit, so a consumer must handle "message present, object missing." Use it when payloads exceed a few hundred kilobytes, or when payload sizes are highly variable (a topic of mostly-2 KB messages with occasional 30 MB ones is the ideal case: send small ones inline and claim-check only the large, deciding by a size threshold in the producer). Do not use it when payloads are uniformly small — you have added a network hop and a lifecycle problem for nothing.

11.5Saga — distributed transactions without distributed transactions#

Plain definition. A saga is a sequence of local transactions across multiple services, where each local transaction publishes an event that triggers the next, and where failure is handled not by rollback but by executing explicit compensating transactions that semantically undo the completed steps.

Why it exists. §11.1 explained why 2PC is unavailable and undesirable across services. But business processes genuinely span services: placing an order touches Order, Payment, Inventory, and Shipping, each with its own database (which is the whole point of microservices, File 10). The saga — from Garcia-Molina and Salem's 1987 paper on long-lived transactions — accepts that you cannot have ACID across services and asks a different question: if you cannot roll back, can you undo? Usually you can, but semantically rather than physically: you cannot un-commit a payment capture, but you can issue a refund; you cannot un-ship a package, but you can issue a return label.

The two coordination styles.

(a) Choreography — no coordinator. Each service subscribes to the events it cares about and publishes its own result. Nobody is in charge.

 Order svc:      OrderCreated ─────────▶
 Payment svc:                  (on OrderCreated) charge → PaymentCaptured ─────────▶
 Inventory svc:                                  (on PaymentCaptured) reserve → StockReserved ──▶
 Shipping svc:                                                        (on StockReserved) ship → Shipped

Pros: maximum decoupling — adding a Loyalty service that also reacts to PaymentCaptured requires touching nothing that exists. No single point of failure. Genuinely simple for 2–3 steps. Cons: the process exists nowhere in the code. To answer "what happens when an order is placed?" you must read four services and hold the graph in your head. Cyclic dependencies creep in. Debugging requires distributed tracing because there is no place to look. And the compensation logic is scattered across every service. Use for: short sagas (≤ 4 steps) with stable, simple flows.

(b) Orchestration — an explicit coordinator. A saga orchestrator (its own service, or a workflow engine like Temporal, AWS Step Functions, Camunda, or Netflix Conductor) holds the state machine and sends commands to participants, receiving replies.

              ┌──────────────────── Order Saga Orchestrator ────────────────────┐
              │  state: AWAITING_PAYMENT → AWAITING_STOCK → AWAITING_SHIPMENT   │
              └───┬──────────▲──────────┬──────────▲──────────┬──────────▲──────┘
      ChargeCard  │          │ Captured │ Reserve  │ Reserved │ Ship     │ Shipped
                  ▼          │          ▼          │          ▼          │
              [ Payment ]────┘      [ Inventory ]──┘      [ Shipping ]───┘

Pros: the business process is one readable state machine in one place; the current state of every in-flight saga is queryable ("show me all orders stuck awaiting stock"); timeouts and retries per step are centralised; compensation is explicit and testable. Cons: the orchestrator is a component to build, deploy, and scale, and a naive one becomes a god-object that knows too much about every domain. Use for: anything with more than 4 steps, anything with real money, anything you will be asked to report on. Decision rule: choreography for simple, stable, short flows; orchestration the moment the flow branches, needs timeouts, or someone asks "where is order 881 right now?"

A worked multi-step example, including a failure and its compensation. Order ord_881, $99.00, 1× SKU WIDGET-7.

 STEP  SERVICE     LOCAL TRANSACTION                      EVENT EMITTED        COMPENSATION IF LATER FAILS
 ────  ─────────   ────────────────────────────────────   ──────────────────   ─────────────────────────────
  1    Order       INSERT orders(status='PENDING')        OrderCreated         UPDATE status='CANCELLED'
  2    Payment     INSERT payments(auth, $99, captured)   PaymentCaptured      Refund $99 (new ledger row)
  3    Inventory   UPDATE stock SET qty=qty-1             StockReserved        UPDATE stock SET qty=qty+1
                     WHERE sku='WIDGET-7' AND qty>=1                            (release reservation)
  4    Shipping    INSERT shipments(label)                Shipped              Cancel label / issue return
  5    Order       UPDATE orders(status='CONFIRMED')      OrderConfirmed       —

Now the failure. Step 3 fails: UPDATE ... WHERE qty >= 1 affects 0 rows — the warehouse's physical count was wrong and WIDGET-7 is actually out of stock. Inventory emits StockReservationFailed{order_id: ord_881, reason: OUT_OF_STOCK}.

 t0   Inventory emits StockReservationFailed
 t1   Orchestrator moves saga ord_881 to state COMPENSATING and walks the completed
      steps in REVERSE order:
 t2   → command RefundPayment{order_id: ord_881, amount: 99.00,
                              idempotency_key: "saga-ord_881-compensate-payment"}
      Payment service does NOT delete the capture row -- that would destroy the audit
      trail and is impossible anyway once the money moved. It inserts a NEW refund
      row. The ledger now shows: +$99 captured, −$99 refunded. Net zero, fully audited.
      This is what "semantic" undo means, and it is why compensation is not rollback.
 t3   → command CancelOrder{order_id: ord_881, reason: OUT_OF_STOCK}
      Order service sets status='CANCELLED'.
 t4   Saga terminates in state COMPENSATED. Customer is emailed an apology.

Three details are load-bearing. Compensations run in reverse order of the forward steps, because later steps may depend on earlier ones. Every command carries an idempotency key (saga-ord_881-compensate-payment) because the orchestrator's delivery to the participant is at-least-once — the orchestrator can crash after sending and re-send on recovery, and a refund executed twice is a real financial loss (File 15 is the mechanism). Compensations must themselves be retried until they succeed, and therefore must be designed to always be able to succeed — a compensation that can permanently fail leaves the saga stuck, which is why "cancel the shipment" is usually implemented as "issue a return label" rather than "recall the truck." Steps that genuinely cannot be compensated (an email was sent, a physical package left the building) must be ordered last in the saga; the standard discipline is compensatable steps first, then a pivot step, then retriable-only steps after it.

The isolation anomalies sagas permit — the part that separates a good answer from a great one. A saga is Atomic (eventually all-or-compensated), Consistent, and Durable, but it is emphatically not Isolated: its intermediate states are visible to everyone. Three named anomalies follow, and you must design against each:

The fourth countermeasure, applicable to all three, is by-value routing: for high-value or high-risk requests, do not use a saga at all — route them to a path that holds a real distributed lock or human approval, and accept the throughput cost for the 0.1% of requests where an anomaly would be catastrophic.

Where sagas connect to the rest of this file. Every step transition is a message (§2.1); every command and event needs at-least-once delivery plus idempotency (§5.2, §6.7); the orchestrator's own state must be persisted before it sends a command, which is a dual write solved by an outbox (§11.2); and a step that never replies must be caught by a timeout, which is why orchestrators need scheduled/delayed messages — a capability Kafka does not have natively (§2.12) and RabbitMQ, SQS, and dedicated workflow engines do (§14.6).

11.6Fan-out and fan-in shapes#

These are the two composition patterns you will draw on a whiteboard, and they have real failure modes rather than being mere diagrams.

Fan-out — one event, many independent consumers. The producer publishes once; N consumers each process the event for their own purpose. In a log (§2.9) this is free: N consumer groups read the same physical bytes, adding read bandwidth (N × the topic's throughput) and zero storage. In a traditional pub/sub it is not free: SNS delivers a physical copy into each of N SQS queues, so storage and per-message cost scale with N. The failure mode of fan-out is the slowest consumer defines nothing and the fastest defines nothing — they are genuinely independent, which is the point, but it means one consumer group being 6 hours behind is invisible unless you monitor per-group lag (§6.9), and it means retention must be sized for the slowest group, not the average.

Fan-out with filtering. When most consumers want a small subset, a dumb broker forces each to read 100% and discard 99% (§2.12). Three real answers: publish to separate topics by type (best when the taxonomy is stable and small); use a broker with server-side routing — RabbitMQ's topic exchange, SNS message-filter policies, EventBridge content-based rules (§14.2, §14.3) — which is precisely what those products are for; or accept the read amplification, which is fine at low volume and ruinous at high.

Fan-in — many producers, one logical stream. Many services write to one topic, or many partitions feed one aggregating consumer. Two failure modes to name. First, there is no ordering across producers: Kafka orders records within a partition, so events from producers A and B interleave arbitrarily even with identical timestamps — any consumer that assumes causality between two different producers' events is wrong, and the fix is either a shared partition key (making them co-ordered) or explicit causality metadata (a correlation ID plus a sequence number, or a version vector). Second, schema divergence: fifteen producers writing "the same" event to one topic will, without enforcement, write fifteen dialects. The mechanism that prevents this is a schema registry — a service holding versioned schemas (Avro, Protobuf, JSON Schema) where the producer's serializer registers the schema and embeds a small schema ID in each message, the consumer's deserializer fetches the schema by ID, and the registry rejects a new schema version that is not compatible with the configured policy (BACKWARD: new consumers can read old data; FORWARD: old consumers can read new data; FULL: both). Confluent Schema Registry, AWS Glue Schema Registry, and Apicurio are the implementations. Without one, fan-in topics rot within a year, and the failure is discovered by a consumer crashing on a field that quietly changed type.

Scatter-gather is fan-out followed by fan-in: dispatch a request to N workers, collect N replies, aggregate. Its distinctive problem is completion detection — you must decide what to do when only 7 of 10 replies arrive, which requires a timeout, a partial-result policy, and a correlation ID on every message so replies can be matched to their request. This is the shape underneath a distributed search query (File 13) and any "ask every shard" pattern (File 04).

Tie-back. All four patterns in this section exist because the log solved transport and left the seam between transport and state untouched. The outbox and CDC close that seam on the write side; idempotency (File 15) closes it on the read side; sagas extend it across services; and claim-check keeps the payloads that would otherwise destroy the broker outside it. Interviewers ask about the dual-write problem constantly, and "I'd use the transactional outbox, or CDC if I already run Debezium, and make consumers idempotent because either way it's at-least-once" is a complete, senior answer in one sentence.

12Pros, Cons & Trade-offs#

12.1Benefits#

12.2Costs & Trade-offs#

13Interview Diagnostic Framework (Symptom → Solution)#

14Real Implementations & Product Landscape#

Every section so far has taught mechanism. This section is the catalogue: the actual software and services that implement those mechanisms, what each one is, what architectural choice distinguishes it from its siblings, what it genuinely costs, and — the part that matters in an interview and on the job — which one you should pick in which situation. When this file (or an interviewer) says "put it on a queue," this is the section that makes the sentence mean something, because "Kafka," "RabbitMQ," and "SQS" are three products that solve overlapping but genuinely different problems, and choosing the wrong one is not a preference, it is an architecture you will fight for years.

The landscape divides into five families. The first three map onto §3's queue-versus-log distinction; the fourth is the lightweight tier people reach for too early or too late; the fifth is what runs on top of the others.

14.1Family 1 — Log-based streaming platforms#

These implement the §3.2 model: a partitioned, replicated, append-only log with consumer-managed offsets, retention independent of consumption, and replay.

Apache Kafka. Built at LinkedIn in 2010 by Jay Kreps, Neha Narkhede, and Jun Rao to solve a specific, real problem described in §15.1 — LinkedIn had a mess of point-to-point pipelines between databases, search, analytics, and monitoring, and needed one durable, high-throughput backbone. Donated to Apache in 2011; Confluent was founded in 2014 by the same people to commercialise it. Its defining architectural choice is the dumb broker (§2.12): the broker maintains no per-message state, appends batches sequentially to disk, serves them with sendfile zero-copy from the page cache (§4.5), and pushes all bookkeeping onto the consumer. Everything Kafka is good at follows from that one decision. What it is genuinely best at: sustained high throughput (hundreds of thousands to millions of messages/second per cluster on commodity hardware), long retention with replay, and being the shared backbone that many independent consumer groups read at their own pace for free (§2.9). Its distinctive capabilities in the current version: KRaft (§4.7), which replaced ZooKeeper with an embedded Raft controller quorum — this removed an entire separate distributed system from the operational surface, cut partition-leader-failover time from minutes to seconds at high partition counts, and raised the practical partition ceiling into the millions; EOS (exactly-once semantics, §5.2) via idempotent producers and transactions; log compaction (§9.3); and tiered storage (§9.4, KIP-405, GA in 3.6). Its limits, stated honestly: it cannot do per-message routing, per-message TTL, priority, or delayed delivery — the features §2.12 said the dumb-broker inversion costs you; it has no per-message acknowledgement and no nack (§6.7), so a single poison message head-of-line blocks its partition (§7.3); consumer parallelism is capped at the partition count; and it is genuinely heavy to operate — JVM tuning, partition-count planning, rebalance behaviour, ISR health, and disk sizing are all things you must actually understand, which is what the previous fourteen thousand words were for. Cost: free and open source; managed as Confluent Cloud (which prices per GB ingress/egress plus storage plus a per-cluster hourly rate, and where a modest production cluster lands in the low thousands of dollars per month) or AWS MSK (priced per broker-hour plus storage — e.g. three kafka.m5.large brokers ≈ $0.21/hour each ≈ $460/month plus EBS, and MSK Serverless prices per GB instead). Reach for it when: you need replay, many independent consumers of one stream, high sustained throughput, or stream processing. Not when: you have 200 messages a day and needed a work queue.

Apache Pulsar. Built at Yahoo around 2013 and open-sourced in 2016, Pulsar's defining choice is separating compute from storage: a Pulsar broker is stateless — it serves clients, owns topic ownership, and holds no data — while durability lives in Apache BookKeeper, a separate replicated log-storage service whose storage nodes are called bookies. A topic's data is written as a sequence of ledgers, each striped across a configurable ensemble of bookies with a write quorum and an ack quorum (e.g. E=3, Qw=3, Qa=2 means each entry goes to 3 bookies and is acknowledged when 2 confirm — note this is a genuine quorum protocol, unlike Kafka's ISR, §8.6). What that buys, concretely: (1) instant broker failover and scaling — because brokers hold no data, a dead broker's topics are reassigned in milliseconds with zero data movement, whereas adding a Kafka broker requires physically rebalancing partitions across the network, which can take hours for a large cluster; (2) independent scaling — add bookies for storage, brokers for connections and throughput; (3) no partition-rebalancing operation at all, which is Pulsar users' most-cited advantage. Its other genuine differentiators: first-class multi-tenancy (a built-in tenant/namespace/topic hierarchy with per-namespace quotas, authentication, retention, and backlog policies — in Kafka, multi-tenancy is a naming convention plus ACLs plus discipline); native geo-replication configured declaratively per namespace, versus Kafka's separate MirrorMaker 2 process; four subscription types that span the queue/log divide in one product — Exclusive, Failover, Shared (round-robin per message across consumers, i.e. genuine queue semantics with per-message ack, which Kafka simply cannot do), and Key_Shared (per-message distribution but with key affinity preserved); per-message TTL and delayed delivery; and mature tiered storage since 2018. Its limits: you are operating two distributed systems plus a ZooKeeper/etcd metadata store instead of one, which is a real and frequently-underestimated operational cost; the ecosystem — connectors, client-library maturity, operational tooling, and the pool of engineers who have run it — is a fraction of Kafka's; and its Kafka-protocol compatibility layer (KoP) is a compatibility shim, not a drop-in. Reach for it when: you need many tenants on one platform, you need both queue and log semantics without running two products, you need geo-replication as a first-class feature, or your workload has topic counts in the hundreds of thousands (Pulsar handles this far better than Kafka). Not when: your team is small and Kafka's ecosystem is what you can hire for.

Redpanda. Founded in 2019, Redpanda is a complete rewrite of a Kafka-API-compatible broker in C++ with no JVM and no ZooKeeper, using a thread-per-core (shard-per-core) architecture built on the Seastar framework: each CPU core owns a slice of the data and its own memory, and cores communicate by explicit message passing rather than shared memory with locks. Why that matters: it eliminates JVM garbage-collection pauses (a direct cause of ISR flapping and rebalance storms in Kafka, §6.6, §8.4), eliminates lock contention, and uses direct I/O (io_uring, bypassing the page cache Kafka depends on) with its own scheduling — which yields consistently lower tail latency (p99.9 in single-digit milliseconds where Kafka's is often tens or hundreds) and 2–3× the throughput per core on the same hardware. It replicates with Raft per partition (genuine consensus, unlike ISR — §8.6), and it is a single binary with no external dependencies. What it is best at: low, predictable tail latency; drastically simpler operations; and getting Kafka semantics on much smaller hardware. Its limits: it is a single-vendor open-core product (the free tier is source-available under the Business Source License, with enterprise features paid); the ecosystem is Kafka's by compatibility rather than its own; and any Kafka feature is available only once Redpanda implements it. Reach for it when: you want the Kafka API but the JVM's operational profile is unacceptable — latency-sensitive workloads, small teams, edge deployments. Not when: you need something specific from the Kafka/Confluent ecosystem that the compatibility layer does not cover.

AWS Kinesis Data Streams. Amazon's managed streaming service, and a genuine log — ordered, replayable, with consumer-managed positions. Its architecture: the unit is the shard (Kinesis's word for a partition), each admitting 1 MB/s or 1,000 records/s of writes and 2 MB/s of reads, and you scale by splitting and merging shards. Records are assigned to shards by an MD5 hash of the partition key into a 128-bit hash-key space, and a shard owns a contiguous range of that space — which means splitting a shard is a range split, not a rehash, so unlike Kafka increasing capacity does not break key→shard stability for keys outside the split range (a real advantage over §2.7's "you can never decrease partitions and increasing them breaks ordering"). Retention is 24 hours by default, up to 365 days. Enhanced fan-out gives each consumer its own dedicated 2 MB/s per shard via an HTTP/2 push subscription rather than sharing the shard's read budget, at extra cost. Pricing: provisioned mode is ~$0.015 per shard-hour (~$11/shard/month) plus $0.014 per million PUT payload units (25 KB each); on-demand mode charges per GB. Best at: being already there if you are on AWS — IAM for auth, no cluster to run, native integration with Lambda, Firehose, and Kinesis Data Analytics. Limits: the hard per-shard throughput ceilings mean capacity planning is shard-count planning; there is no compaction; retention maxes at a year; the client-side story is weaker (the Kinesis Client Library uses a DynamoDB table for lease coordination and checkpoints, which is another moving part with its own throttling failure mode); and cost at high throughput exceeds a self-run Kafka cluster by a wide margin. Reach for it when: you are on AWS, throughput is moderate, and eliminating operations matters more than cost or features.

Azure Event Hubs. Microsoft's managed streaming service, architecturally very close to Kafka — partitions, offsets, consumer groups, retention — and it exposes a Kafka-protocol-compatible endpoint, so most Kafka clients work against it unchanged (with EventHubs-flavoured auth). Capacity is sold in Throughput Units (1 TU = 1 MB/s or 1,000 events/s ingress, 2 MB/s egress) in Standard tier, or Processing Units in the Premium/Dedicated tiers. Event Hubs Capture automatically writes the stream to Blob Storage or Data Lake in Avro — tiered storage as a checkbox. Best at: Azure-native shops wanting Kafka semantics without a cluster. Limits: partition count is fixed at creation in Standard tier, retention is 1–7 days (up to 90 in Premium), no compaction, and Kafka compatibility does not extend to transactions or the admin API in full.

**Google Cloud Pub/Sub — and why it is not a Kafka-style log. This distinction is worth stating carefully because the name misleads. Pub/Sub is a globally-distributed, per-message-acknowledged pub/sub system in the §2.9 sense: you publish to a topic, each subscriber creates a subscription which holds its own independent backlog, and the service tracks per-message acknowledgement state with an ack deadline (§2.10) — the visibility-timeout model, not the offset model. There are no partitions you manage and no offsets: it autoscales without capacity planning, which is genuinely excellent, but it also means default ordering is not guaranteed at all unless you publish with an ordering key (which then constrains throughput for that key to one publisher region), and replay is not "seek to an offset" — it is Seek, which rewinds a subscription to a timestamp or a named snapshot**, within a configured message_retention_duration (default 7 days, max 31). Pub/Sub Lite was the Kafka-shaped, partition-and-capacity-managed cheaper variant — note that it has been deprecated, and the current answer for Kafka semantics on GCP is Google's managed Kafka offering or self-hosting. Best at: globally-distributed, zero-ops, wildly-variable-throughput event delivery — it will absorb a 100× spike without you doing anything, which no partitioned system will. Limits: ordering is awkward, there is no compaction, no log semantics, and no consumer-lag-style offset arithmetic; you reason about backlog in unacked-message counts and oldest-unacked-age instead. Pricing: ~$40/TiB of throughput, which is simple and gets expensive at scale.

14.2Family 2 — Traditional message brokers#

These implement §3.1: per-message state in the broker, per-message acknowledgement, server-side routing, and deletion on ack. This is the "smart broker" side of §2.12's inversion.

14.2.1 RabbitMQ#

What it is. Written in Erlang and released in 2007, RabbitMQ is the reference implementation of AMQP 0-9-1 (Advanced Message Queuing Protocol) — an open wire protocol designed by a consortium of financial firms who wanted broker interoperability. Erlang is not incidental: the language's lightweight processes, supervision trees, and built-in distribution gave RabbitMQ clustering and fault tolerance almost for free, and are why one node comfortably manages hundreds of thousands of concurrent queues.

Its distinguishing mechanism — the exchange and the binding. This is the single most important thing to understand about RabbitMQ and the thing Kafka fundamentally cannot do. Producers never publish to a queue. They publish to an exchange, with a routing key (a string). The exchange holds no messages; it is a routing function. Bindings connect an exchange to queues, each binding carrying a pattern. The exchange evaluates its type's rule against the routing key and the bindings, and copies the message into every matching queue. Four exchange types exist and each is a different routing algorithm:

Queue types — quorum versus classic mirrored, and why the change happened. Historically, HA in RabbitMQ meant classic mirrored queues: a master queue on one node with mirrors on others, kept in sync by a custom replication protocol. It had a bad failure mode — during a network partition, mirrors could diverge, and the recovery policies (autoheal, pause_minority, ignore) forced you to choose between availability and losing messages when the partition healed, because reconciliation was "pick a winner and discard the loser's messages." Mirrored queues are deprecated and removed in RabbitMQ 4.x. The replacement is the quorum queue, which replicates using Raft (File 18) — a message is confirmed once a majority of replicas have it, leader election is Raft's, and there is no divergence to reconcile because Raft does not permit it. The trade-offs are explicit: quorum queues are safer and more predictable under partition, but they use more memory and disk per message, have higher latency per publish (a quorum round trip), require an odd replica count (typically 3 or 5), and deliberately do not support some classic-queue features — notably per-message priority and non-durable/transient modes. There is also the stream queue type (RabbitMQ 3.9+), which is an append-only, replayable, non-destructive log inside RabbitMQ — Kafka's model bolted on for the cases where you want both, though without Kafka's throughput.

What it is genuinely best at. Complex per-message routing, per-message acknowledgement and redelivery, priority queues (x-max-priority), delayed/scheduled delivery (via the delayed-message-exchange plugin or a TTL-plus-dead-letter-exchange trick), per-message TTL, and having many thousands of queues with modest traffic each — a task-distribution shape that Kafka is structurally wrong for. Its management UI and per-queue metrics are excellent.

Its limits. Throughput per node is one to two orders of magnitude below Kafka's for durable messages (tens of thousands/second, not millions) — and §2.12 explained exactly why: per-message state is per-message work. Messages are deleted on ack, so there is no replay and no adding a new consumer that sees history (unless you use streams). Deep queues are a performance cliff rather than a normal state: RabbitMQ is designed for queues that stay near-empty, and a queue with ten million messages will page to disk and slow dramatically — the exact opposite of Kafka, where a large backlog is unremarkable. Cost: free and open source; managed as CloudAMQP, Amazon MQ for RabbitMQ, or VMware's commercial distribution.

14.2.2 ActiveMQ and ActiveMQ Artemis#

Apache ActiveMQ ("Classic") is the long-standing Java broker built around JMS (Java Message Service) — not a wire protocol but a Java API specification for messaging, which is the key to understanding its niche: JMS is what enterprise Java applications were written against for twenty years, so ActiveMQ is what you find inside large Java estates. It supports JMS queues and topics, durable subscriptions, message selectors (SQL-like filters evaluated by the broker), transactions, and speaks OpenWire, AMQP, MQTT, and STOMP. ActiveMQ Artemis is the modern successor (donated from HornetQ), with a completely rewritten non-blocking asynchronous core, a journal-based persistence engine that is dramatically faster, and it is the engine inside Red Hat AMQ and the default in Amazon MQ for ActiveMQ. Best at: JMS compatibility for existing Java applications, and enterprise integration where the requirement literally says "JMS 2.0." Limits: far lower throughput than a log; clustering and HA are more manual than RabbitMQ's quorum queues; and for greenfield work there is rarely a reason to choose it over RabbitMQ unless JMS is the requirement. Reach for it when: you are integrating with existing JMS applications or migrating an IBM MQ / WebLogic estate.

14.3Family 3 — Cloud-managed queues#

14.3.1 Amazon SQS#

What it is. Launched in 2004, SQS is AWS's oldest service and the archetype of the fully-managed queue: there is no cluster, no capacity to provision, no connections to manage, and effectively no throughput ceiling on the standard flavour. You SendMessage, ReceiveMessage, and DeleteMessage over HTTPS. Its state model is §2.10's visibility timeout end to end.

Standard versus FIFO — the two products behind one name.

Standard queues offer unlimited throughput, at-least-once delivery, and best-effort ordering — meaning ordering is not guaranteed at all and duplicates genuinely occur (SQS replicates messages across multiple servers, and occasionally a delete does not reach every copy). Consumers must be idempotent, full stop. Cost is $0.40 per million requests after a 1-million-request free tier, and note that a request is a request: a ReceiveMessage returning zero messages still costs one, which is why long polling (WaitTimeSeconds up to 20) is not merely a latency optimisation but a substantial cost optimisation over short polling, and why MaxNumberOfMessages up to 10 per receive matters. Message size is capped at 256 KB (use claim-check, §11.4, or the SQS Extended Client Library which does it for you), retention is 4 days by default, 1 minute to 14 days configurable, and in-flight messages are capped at 120,000 (§6.8).

FIFO queues (names must end in .fifo) add strict ordering and exactly-once processing within a scope, via two required concepts. **MessageGroupId is the ordering and parallelism unit — it is SQS's partition key. Messages sharing a group ID are delivered in strict order, one group is only ever in flight to one consumer at a time, and different groups are processed in parallel**. So the group ID choice is exactly the §2.7 partition-key trade-off: one group ID for everything means total ordering and no parallelism; order_id as the group ID means per-order ordering with unlimited parallelism. **MessageDeduplicationId gives a 5-minute deduplication window**: a message with the same dedupe ID sent twice within 5 minutes is accepted once and silently discarded the second time. You can supply the ID explicitly (best — use your business idempotency key) or enable ContentBasedDeduplication, which SHA-256s the message body. Throughput limits are the thing to know: FIFO gives 300 transactions/second per API action (3,000 messages/second when batching 10 per call), or up to 70,000 messages/second in high-throughput mode, which relaxes the per-group serialisation constraints and requires many distinct group IDs. FIFO costs $0.50 per million requests.

Best at: decoupling AWS services with zero operational burden; absorbing spikes; and per-message work distribution where each item is independent. Limits: no replay whatsoever (a consumed and deleted message is gone — this is the single biggest difference from Kafka), no fan-out on its own (that is SNS's job), 14-day maximum retention, per-message API cost that becomes significant at high volume, and no server-side filtering. Native features worth naming: the redrive policy (maxReceiveCount → automatic DLQ, §2.10, §7.2), DLQ redrive to move messages back to the source queue after a fix, and **DelaySeconds** (up to 15 minutes) for delayed delivery.

14.3.2 Amazon SNS and the SNS→SQS fan-out pattern#

SNS (Simple Notification Service) is a topic in the §2.9 sense: a routing point that stores nothing. You publish once; SNS delivers a copy to every subscription, where a subscription's endpoint can be an SQS queue, a Lambda function, an HTTPS endpoint, email, SMS, or a mobile push service.

The canonical pattern, and why it exists. SQS alone is point-to-point: N consumers on one queue compete for messages, so you cannot have three services each independently process every order by sharing one queue. SNS alone has no durable backlog: if a subscribed HTTPS endpoint is down past the retry policy, the message is lost. Composing them gives you Kafka-shaped fan-out with queue-shaped durability:

                       ┌────────▶ SQS: shipping-queue    ──▶ shipping service
   OrderService ──▶ SNS ────────▶ SQS: analytics-queue   ──▶ analytics service
     (publish once)    └────────▶ SQS: email-queue       ──▶ email service
                        (SNS copies into each queue; each queue has its OWN
                         backlog, its OWN DLQ, its OWN retry, its OWN consumers)

Each consumer is fully independent — one being down for six hours simply grows its own queue, and the others are unaffected. The cost model is the crucial contrast with a log (§2.9): every subscription creates a physical copy, so N subscribers cost N× the storage and N× the SQS request charges, whereas N Kafka consumer groups cost zero extra storage. Adding a tenth subscriber to SNS multiplies cost; adding a tenth consumer group to Kafka does not.

**RawMessageDelivery is a subscription setting you almost always want on: without it, SNS wraps your payload in a JSON envelope that every consumer must unwrap. Subscription filter policies** are SNS's server-side filtering: a JSON policy on the subscription (matching on message attributes, or on the body with MessageBody filtering) so a subscriber receives only the subset it cares about, without paying to receive and discard the rest — the routing capability §11.6 said a dumb broker cannot offer. FIFO topics exist and can only fan out to FIFO queues. Pricing: $0.50 per million publishes, plus the delivery cost per endpoint type, plus the downstream SQS charges.

14.3.3 Amazon EventBridge#

What it is. EventBridge (formerly CloudWatch Events) is an event bus with content-based routing rules: events arrive on a bus, and rules with JSON pattern matching over the event body route matches to targets (Lambda, SQS, Step Functions, Kinesis, API Destinations, another bus). Its distinguishing capabilities are the pattern language — prefix matching, numeric ranges, anything-but, existence checks — which is far richer than SNS's filter policies; schema discovery and a registry that infers schemas from observed events and generates typed bindings; SaaS partner event sources (Datadog, Shopify, Zendesk publishing directly into your bus); archive and replay of events within a retention window; and EventBridge Pipes for point-to-point source→enrich→target flows, plus Scheduler for cron-like and one-off scheduled events. Best at: low-to-moderate-volume, high-variety application and SaaS integration where the routing logic is the hard part. Limits: throughput and latency are modest by streaming standards (thousands of events/second per bus by default, and latency in the hundreds of milliseconds), the price is $1.00 per million events — 2.5× SQS and vastly more than Kafka at volume — and it is not a stream: replay exists but ordered offset-based consumption does not. Decision rule: EventBridge when the value is in routing and integration; Kinesis or Kafka when the value is in volume and replay; SNS+SQS when you just need durable fan-out cheaply.

14.4Family 4 — Lightweight and embedded#

NATS and JetStream. NATS Core, created by Derek Collison in 2010 and now a CNCF project, is a deliberately minimal messaging system: a single small Go binary, a text-based protocol, subject-based addressing with wildcards (orders.eu.*, orders.>), and at-most-once, fire-and-forget delivery with no persistence at all. That last point is the design, not a limitation: NATS Core will drop messages for a slow consumer rather than buffer them, which is why it achieves millions of messages/second with microsecond latencies. It also natively supports request–reply (with an auto-generated inbox subject) and queue groups (competing consumers on a subject), which makes it an excellent internal service-to-service bus. JetStream is the persistence layer added in 2020: it introduces streams (durable, replayable, with retention by limits, interest, or work-queue semantics), consumers (durable or ephemeral, push or pull, with explicit ack, ack-wait redelivery, and max-deliver limits leading to a DLQ), Raft-based replication (R=1/3/5), exactly-once via message deduplication on a Nats-Msg-Id header, and a key-value and object store built on top of streams. Best at: low-latency internal messaging, edge and IoT deployments (the binary is ~15 MB and runs anywhere, with built-in leaf-node topology for edge-to-cloud), and being radically simpler to operate than Kafka while covering a large fraction of the use cases. Limits: a much smaller connector and stream-processing ecosystem, lower per-stream throughput than Kafka at the very top end, and a smaller pool of operational experience. Reach for it when: you want durable streaming without a Kafka-sized operational commitment, or when latency is measured in microseconds, or when you deploy to the edge.

Redis Streams. Added in Redis 5.0 (2018), a Redis Stream is a genuine append-only log data structure living inside Redis, with entry IDs of the form <milliseconds>-<sequence> that are monotonically increasing and act as offsets. The commands, individually: **XADD key * field value ...** appends an entry and returns its ID (the * asks Redis to generate the ID; you can cap the stream with MAXLEN ~ 1000000, where ~ means approximate trimming at node boundaries and is much cheaper than exact). **XREAD reads entries after a given ID — the simple, non-grouped consumption path. XGROUP CREATE key groupname $** creates a consumer group ($ = start from now, 0 = from the beginning). **XREADGROUP GROUP g c COUNT 10 BLOCK 5000 STREAMS key >** reads new entries for consumer c in group g — and crucially, entries delivered this way are recorded in the PEL (Pending Entries List), the per-group set of delivered-but-unacknowledged entries, which is Redis's equivalent of in-flight messages. **XACK key g id removes an entry from the PEL, completing it. XPENDING inspects the PEL (how many, whose, how long ago), and XCLAIM / XAUTOCLAIM transfer ownership of entries pending longer than a minimum idle time to another consumer — this is how you recover work from a dead consumer, and it is the visibility-timeout model (§2.10) implemented by hand. Best at: giving you real consumer groups, per-message ack, and replay when Redis is already in your stack and your volume is modest. Limits, stated plainly: the stream lives in RAM**, so retention is bounded by memory and you must cap it with MAXLEN; durability is Redis's, which means AOF with appendfsync everysec loses up to a second of writes on a crash, and asynchronous replication means a failover can lose acknowledged writes (§8.2's acks=1 problem, in a different product); there is no partitioning of a single stream across nodes, so one stream's throughput is one node's throughput and Redis Cluster shards keys, not a single stream; and there is no compaction. Reach for it when: you need a durable work queue or modest event stream, Redis is already deployed, and the data is reconstructible. Do not reach for it when the stream is a system of record.

Redis Pub/Sub — the trap. This deserves its own explicit warning because it is the most common serious mistake in this entire landscape. Redis's PUBLISH / SUBSCRIBE commands are fire-and-forget with absolutely no durability. The mechanism is brutally simple: PUBLISH walks the list of clients currently subscribed to the channel and writes the message to each socket. That is all. If no subscriber is connected, the message is discarded and nothing records that it existed. If a subscriber is connected but slow, it is disconnected when its output buffer exceeds client-output-buffer-limit pubsub (default 32 MB hard / 8 MB over 60 s soft) and the messages it had not read are gone. There is no acknowledgement, no offset, no replay, no persistence to AOF or RDB, and no delivery across a Redis failover — and messages published on a replica are not propagated to the primary's subscribers. It is at-most-once with an unreliable "most." File 02 §2.9 covers it in the caching context, where it is used for cache-invalidation broadcasts and where a missed invalidation is usually survivable. Everywhere else it is a trap, and the specific way it bites is that it works perfectly in development and in staging, and drops messages in production during exactly the events you built the system to survive: a deploy, a restart, a network blip, a GC pause. If a message must not be lost, Redis Pub/Sub is disqualified — use Redis Streams, which was added to Redis precisely because so many people had made this mistake.

14.5Family 5 — Stream processors#

These are not brokers; they are the frameworks that implement §10 on top of a broker.

Kafka Streams. A Java library, not a cluster — this is its defining property. You add a dependency, write a topology, and run your application as an ordinary JVM process (a container, a Kubernetes Deployment, a JAR); parallelism comes from running more instances, which coordinate through the ordinary consumer-group protocol of §6. There is no scheduler, no job manager, no separate cluster to operate. It gives you the DSL (map, filter, groupByKey, aggregate, windowing, joins), KTable/KStream duality (a stream of updates is a table, and a table's changelog is a stream — the idea §9.3's compaction makes physical), RocksDB state stores with compacted changelogs (§10.6), interactive queries, and exactly_once_v2 (§10.7). Limits: Kafka in, Kafka out — it cannot read from Kinesis or write to a database as a first-class source/sink (you bridge via Kafka Connect); JVM only; and it inherits the consumer group's rebalance behaviour. Reach for it when: your input and output are Kafka and you want stream processing without operating a second platform. This is the right default for most teams.

Apache Flink. A true distributed stream-processing engine with its own cluster (JobManager + TaskManagers), originating from the Stratosphere research project in Berlin and now the industry's most capable streaming runtime. Its differentiators over Kafka Streams: many sources and sinks (Kafka, Kinesis, Pulsar, files, JDBC, Iceberg, Elasticsearch); true event-time semantics with explicit watermarks (§10.3) rather than Kafka Streams' stream-time approximation; checkpointing and savepoints (§10.7) — a savepoint is a manually-triggered, self-contained snapshot you can use to stop a job, deploy new code, and resume from exactly where it was, which is the single most valuable operational feature in streaming; sophisticated CEP (complex event processing) for pattern detection over sequences; Flink SQL with the unified batch/stream semantics that let the same query run over history and over a live stream; and far better handling of very large state. Limits: you operate a Flink cluster (or pay for Amazon Managed Service for Apache Flink, Confluent's Flink, or Ververica); the learning curve is genuinely steep; and it is overkill for a simple map+filter pipeline. Reach for it when: you need event-time correctness, multiple systems in one pipeline, very large state, savepoint-based deployment, or CEP.

Spark Structured Streaming. Streaming expressed as an incremental query over an unbounded table, executing on the Spark engine. Historically it was micro-batch — collecting records for an interval (default ~100 ms to seconds) and running a small batch job — which gives high throughput and excellent integration with Spark's batch, SQL, and MLlib ecosystem, at the cost of latency floors in the hundreds of milliseconds; a continuous-processing mode with ~1 ms latency exists but supports a restricted set of operations. Reach for it when: you already run Spark, the same team writes batch and streaming jobs over the same data (especially on Delta Lake / Iceberg lakehouse storage), and second-scale latency is fine. Not when: you need millisecond latency or fine-grained event-time control — Flink is better at both.

ksqlDB. Confluent's SQL layer over Kafka Streams: you write CREATE STREAM/CREATE TABLE and continuous SELECT queries, and a ksqlDB server cluster runs them as Kafka Streams topologies underneath. Best at: letting analysts and non-Java engineers build streaming transformations, and rapid prototyping. Limits: it is another cluster to run, SQL cannot express everything a topology can, and it is Confluent-centric. Flink SQL is the vendor-neutral competitor and has broader adoption for new work.

14.6Comparison table and decision rules#

Decision rules — say these out loud in an interview.

14.7Anti-recommendations — the common wrong picks and why people make them#

15Real-World Engineering Scenarios#

15.1LinkedIn — The Birthplace of Kafka#

LinkedIn created Kafka to solve data-pipeline chaos:

15.2Uber — Async Everything (dispatch, surge, payments)#

Uber runs enormous Kafka infrastructure:

15.3Payments / E-commerce — Order Processing with SQS + Idempotency + DLQ#

A canonical async order flow (e.g., on AWS):

16Interview Gotchas & Failure Modes#

16.1Classic Questions#

16.2Failure Modes & Mitigations#

17Whiteboard Cheat Sheet#

   PRODUCER ──publish──►  [ BROKER: durable, replicated ]  ──►  CONSUMER GROUP(s)
                                                                 (pull, own offset)

 QUEUE (RabbitMQ/SQS): consume+DELETE. smart broker. task distribution. NO replay.
 LOG  (Kafka/Kinesis): append+RETAIN. dumb broker. offsets. REPLAY. millions/sec.

 KAFKA:
   Topic ──split──► Partitions (parallelism + ordering unit; key -> hash -> partition)
   Partition = ordered append-only log; message has OFFSET (0,1,2...)
   Consumer group: 1 partition -> 1 consumer  (max parallelism = #partitions)
   Different groups each read the FULL stream (pub/sub)
   Replication: leader + followers; ISR (in-sync); new leader elected from ISR
   Durability knob: acks=0(fast,lossy) | acks=1(leader) | acks=all + min.insync.replicas (safe)
   Fast because: sequential disk I/O + zero-copy sendfile + batching + page cache + dumb broker
   Compaction: keep latest value per key -> changelog/materialized table (CDC, event sourcing)

 DELIVERY: at-most-once(lose) | at-least-once(dupe -> NEED IDEMPOTENCY) | exactly-once(processing only)
 ORDER: only within a partition. Key related events together. Global order = 1 partition.
 RESILIENCE: retry+backoff -> cap -> DLQ (poison pill). Backpressure = pull + disk log; watch LAG.

One-sentence summary for an interviewer:

"Message queues and event logs decouple services by putting a durable buffer between them — a queue (consume-and-delete) distributes tasks, while a log like Kafka (append, retain, replay) streams events to many independent consumers at millions/sec; Kafka scales and orders via partitions (one partition→one consumer per group, order only within a partition), never loses data via **replication + ISR + acks=all, and since delivery is at-least-once, consumers must be idempotent to get effectively-once processing — with DLQs for poison pills and consumer-lag** monitoring for backpressure."

End of File 07. Next → File 08: Distributed Consensus & Replication (CAP theorem, quorums, Raft/Paxos, leader election).