Concept 1: Load Balancing

System Design Bible — File 01 of 10 Difficulty: ★☆☆☆☆ (Most Fundamental) Prerequisites: The four foundations unpacked in Section 0 below (TCP/IP, HTTP, client-server, DNS).

Table of Contents#

0Prerequisites — What You Must Understand First#

Load balancing sits on top of six networking ideas. If any is fuzzy, the mechanics later (especially Layer 4 vs Layer 7, and the forwarding modes in §3.2) will feel like memorization instead of understanding. Here they are, from scratch. The last two — how a message is physically wrapped for transmission, and what a network card actually does with the bits — are the ones almost every load-balancing explanation skips, and they are precisely the ones you need in order to understand why Direct Server Return is even possible.

0.1Client–server model and the "request/response"#

The web works on a client–server model: a client (a browser, a phone app, another service) sends a request over the network to a server (a machine running your application), and the server sends back a response. A request is a small message ("GET me the page at /home"); a response is the data ("here's the HTML"). The server does work to produce that response — CPU, memory, maybe a database lookup — and it can only do a finite amount of that work per second before it's saturated. Load balancing exists entirely because one server can't handle all the requests, so we need many servers and something to spread requests across them. Hold this: a request is a unit of work a server must spend resources on.

0.2IP addresses and ports (the "where")#

Every machine reachable on a network has an IP address — a numeric label like 93.184.216.34 that identifies which machine to deliver packets to. On each machine, a port (a number 0–65535) identifies which program/service on that machine (web servers conventionally listen on port 80 for HTTP and 443 for HTTPS). So the pair **IP:port** (e.g., 93.184.216.34:443) is the full address of a specific service on a specific machine. This matters because a Layer 4 load balancer (§5) makes its decisions using only this addressing information — it sees "a connection from this IP:port to that IP:port" and nothing about the actual content. Understanding that IP:port is all Layer 4 knows is the key to the whole L4-vs-L7 distinction.

0.3TCP vs UDP, and what a "connection" is#

TCP (Transmission Control Protocol) is the reliable, ordered, connection-based transport most of the web runs on. Before data flows, the two sides perform a handshake (the famous SYN → SYN-ACK → ACK three-message exchange) to establish a connection; then bytes are delivered reliably and in order; then the connection is closed. A connection is thus a stateful, ongoing conversation between one client and one server — which is why a load balancer must send all packets of the same TCP connection to the same backend (you can't hand packet 1 to server A and packet 2 to server B, or the conversation breaks). UDP is the connectionless, fire-and-forget alternative (no handshake, no guaranteed delivery/order) used for speed-critical, loss-tolerant traffic (video, gaming, DNS). Load balancers handle both, but TCP's "keep a connection pinned to one backend" requirement shapes much of their design.

0.4HTTP and DNS (the "what" and the "phone book")#

HTTP (HyperText Transfer Protocol) is the application-level language spoken over a TCP connection: a structured text request with a method (GET, POST…), a path (/api/users), headers (metadata like Host:, Cookie:, Authorization:), and optionally a body. HTTPS is HTTP wrapped in TLS encryption. The crucial point for load balancing: this rich, meaningful information — URL, headers, cookies — lives inside the connection's payload, and reading it requires decrypting and parsing the HTTP message. A Layer 7 load balancer (§5) does exactly that, which is why it can route by URL or cookie, while a Layer 4 one (which only sees IP:port, §0.2) cannot. Finally, DNS (Domain Name System) is the internet's phone book: it translates a human name (api.example.com) into an IP address. When you point api.example.com at your load balancer's IP, DNS is what delivers every client to the load balancer's front door in the first place — so DNS and load balancing work hand in hand.

0.5Encapsulation — what a "message" physically looks like on the wire (frames, packets, segments)#

This is the prerequisite that unlocks §2.8 and §3.2, and it is almost always skipped. When your browser sends GET /api/users, that text does not travel across the network as text. It travels wrapped in layers of headers, like a letter placed in an envelope, which is placed in a shipping box, which is placed on a labelled pallet. Each network layer adds its own wrapper on the way down the sending machine's stack, and each layer strips its own wrapper on the way up the receiving machine's stack. This wrapping process is called encapsulation, and the un-wrapping is decapsulation.

Here is the actual shape of one HTTP request leaving your laptop, drawn from the outside in:

+---------------------------------------------------------------------------+
| ETHERNET FRAME  (Layer 2 — the data-link layer)                           |
| dst MAC: aa:bb:cc:dd:ee:01   src MAC: 11:22:33:44:55:66   type: 0x0800    |
|  +----------------------------------------------------------------------+ |
|  | IP PACKET  (Layer 3 — the network layer)                             | |
|  | src IP: 192.168.1.20   dst IP: 1.2.3.4   proto: 6 (TCP)   TTL: 64    | |
|  |  +-----------------------------------------------------------------+ | |
|  |  | TCP SEGMENT  (Layer 4 — the transport layer)                    | | |
|  |  | src port: 44321   dst port: 443   seq/ack   flags: PSH,ACK      | | |
|  |  |  +-----------------------------------------------------------+  | | |
|  |  |  | PAYLOAD  (Layer 7 — the application layer)                |  | | |
|  |  |  | "GET /api/users HTTP/1.1\r\nHost: api.example.com\r\n..."  |  | | |
|  |  |  +-----------------------------------------------------------+  | | |
|  |  +-----------------------------------------------------------------+ | |
|  +----------------------------------------------------------------------+ |
| FCS (4-byte checksum trailer)                                             |
+---------------------------------------------------------------------------+

Now the vocabulary, which people use sloppily and which this file will use precisely:

The critical asymmetry — and this is the single most important sentence in this prerequisite: the IP addresses in the packet header are written once by the sender and stay the same for the entire journey across the internet, but the MAC addresses in the frame header are rewritten at every single hop. Your laptop's frame is addressed to your router's MAC, not to the destination server's MAC — your laptop cannot possibly know that server's MAC, and doesn't need to. Your router strips that frame, reads the IP packet inside, decides the next hop, and wraps the packet in a brand-new frame addressed to the next router's MAC. This happens at every hop, perhaps fifteen times, until the last router on the destination's local network builds a final frame addressed to the destination server's actual MAC. The IP packet inside rode the whole way untouched.

Hold this, because DSR (§3.2 C) is entirely built on it: you can change where a frame is physically delivered — by rewriting its destination MAC — without changing a single byte of the IP addresses inside it. The receiving machine will get a packet that still says "this is addressed to the VIP," even though it was hand-delivered to a specific backend's network card. That loophole is the whole trick.

MTU and why encapsulation is not free. Each link has a MTU (Maximum Transmission Unit) — the largest payload a single frame may carry. Standard Ethernet MTU is 1500 bytes, meaning the IP packet inside a frame may be at most 1500 bytes; add the 14-byte Ethernet header and 4-byte FCS and the frame itself is up to 1518 bytes on the wire. Subtract the 20-byte IP header and the 20-byte TCP header and your application gets 1460 bytes of actual data per full-size segment — a number called the MSS (Maximum Segment Size), which the two ends announce to each other during the TCP handshake. Now the consequence for §3.2 D: if a load balancer wraps that already-full 1500-byte packet inside another 20-byte IP header (IP-in-IP tunneling), the result is 1520 bytes, which exceeds the MTU and must either be fragmented (split into two frames, doubling the packet count and creating a new failure mode — lose either fragment and the whole original packet is lost) or dropped with an ICMP "fragmentation needed" message that tells the sender to shrink. This is why every tunneling deployment involves MTU tuning — lowering the backend's advertised MSS to ~1440, or enabling jumbo frames (MTU 9000) inside the datacenter where you control every switch. "Encapsulation overhead" is not an abstract cost; it is these 20 bytes pushing you over a hard limit.

Hold this: frame wraps packet wraps segment wraps your data; IP addresses survive the whole trip, MAC addresses are rewritten every hop, and every extra wrapper eats into a fixed 1500-byte budget.

0.6The NIC — what a network card actually is and what it does with arriving bits#

A NIC (Network Interface Card / Controller) is the hardware that connects a machine to a network — historically a card you plugged into a PCI slot, today almost always a chip integrated on the motherboard, and in a cloud VM a virtual NIC presented by the hypervisor that behaves identically from the operating system's point of view. It matters here for three reasons: it owns the MAC address, it enforces the bandwidth ceiling that pushes you toward horizontal scaling and DSR, and its offload features explain why some load balancers are fast and others are not.

What it physically does. The NIC converts between the bits your operating system holds in memory and the electrical, optical, or radio signals on the medium. On transmit, the kernel hands it a fully-built frame; the NIC serializes it onto the wire and appends the FCS checksum. On receive, it recovers bits from the signal, checks the FCS, discards corrupted frames, and then applies its MAC filter: it looks at the frame's destination MAC address and asks "is this mine?" If the destination MAC is the NIC's own address, or the broadcast address ff:ff:ff:ff:ff:ff, or a multicast group the machine has joined, the frame is accepted; otherwise it is silently discarded in hardware, without ever waking the CPU. (The exception is promiscuous mode, in which the NIC accepts every frame it sees regardless of address — this is what tcpdump and Wireshark enable to capture traffic, and what a network tap or an intrusion detection sensor relies on.)

How an accepted frame reaches your application. The NIC writes the frame into a ring buffer in main memory by DMA (Direct Memory Access) — copying it directly without CPU involvement — and then raises an interrupt to tell the kernel "there is work." The kernel's driver strips the Ethernet header, sees EtherType 0x0800, and hands the IP packet to the IP layer. The IP layer checks the destination IP: is this address configured on any of my interfaces? If yes, it strips the IP header and hands the segment up by protocol number to TCP. TCP looks at the destination port, finds the socket that is listening on it, and appends the payload to that socket's receive buffer, where your application's read() finally picks it up. Every one of those four checks is a place a packet can be dropped, and knowing which check drops it is how you debug load-balancing problems: a frame with a foreign MAC dies at the NIC, a packet for an IP you don't own dies at the IP layer, a segment for a port nobody is listening on gets a TCP RST, and a segment for a healthy socket whose buffer is full triggers a zero-window stall.

Notice the second check carefully, because §2.8 depends on it: **the IP layer accepts a packet if the destination IP is configured on any interface, not necessarily the interface the packet arrived on.** A packet addressed to the VIP that physically arrives on eth0 is accepted just fine if the VIP happens to be configured on lo. That permissiveness is a deliberate Linux behaviour (the "weak host model"), and DSR is built directly on top of it.

The bandwidth ceiling — the physical limit that shapes the whole file. A NIC has a fixed maximum throughput: 1 Gbps on old commodity hardware, 10/25 Gbps typically today, 100 Gbps on high-end datacenter gear. That number is a hard wall, and it is one of the concrete meanings of "you cannot scale a single machine forever" from §1.2. Work it out: at 10 Gbps you can move roughly 1.25 GB/s. If your service streams 5 Mbps video, one 10 Gbps NIC saturates at about 2,000 concurrent viewers — no matter how much CPU or RAM you add. This is also exactly why the response path matters so much in §3.2: in a full-proxy or NAT design, every byte of every response crosses the load balancer's NIC, so the LB's NIC becomes the ceiling on the whole service. DSR exists to take that ceiling away by letting responses leave from the backends' NICs in parallel — twenty backends with 10 Gbps NICs give you 200 Gbps of egress instead of one LB's 10.

Offloads — why the same software is fast on one box and slow on another. Modern NICs perform work in silicon that the CPU would otherwise do per packet. Checksum offload computes the IP/TCP checksums in hardware. TSO/GSO (TCP/Large Segmentation Offload) lets the kernel hand the NIC one 64 KB buffer and have the NIC chop it into 1460-byte segments itself, cutting per-packet CPU cost by a factor of forty. LRO/GRO (Large/Generic Receive Offload) does the reverse on receive, merging many arriving segments into one big buffer before the kernel processes it. RSS (Receive Side Scaling) hashes each arriving flow's 5-tuple and distributes flows across multiple hardware receive queues, each bound to a different CPU core — without it, a single core handles all interrupts and becomes the bottleneck long before the wire does. These are the reason a software load balancer can push line rate on modern hardware; they are also why a load balancer that must modify packets (rewriting headers, terminating TLS) loses some offload benefits and costs more CPU per byte than one that only rewrites a MAC address.

Hold this: the NIC owns the MAC address, filters frames in hardware by destination MAC, accepts packets for any IP the machine owns on any interface, and imposes a fixed bandwidth ceiling that the entire load-balancer forwarding-mode taxonomy exists to work around.

1Architectural Definition & "The Why"#

1.1The Plain Definition#

A load balancer (LB) is a component — hardware appliance, software daemon, or managed cloud service — that sits between clients and a pool of backend servers ("the fleet" or "the backend pool") and distributes incoming network requests across those servers according to a defined policy. Its job is to ensure that no single server is overwhelmed, that capacity is used evenly, and that the failure of any one server is invisible to the end user.

Think of it as the maître d' of a restaurant. Diners (requests) arrive at the door. The maître d' (load balancer) does not seat everyone at table 1. He looks at which tables (servers) are free, which waiter (server) is least busy, and routes each party to the optimal table so the kitchen throughput stays maximal and no waiter collapses.

1.2"The Why" — What Forced Its Invention#

To understand why load balancing exists, you must understand the historical scaling wall that engineers hit.

The era of the single server (1990s): Early web applications ran on one machine. The web server, the application logic, and often the database all lived on a single box. This works beautifully until it doesn't. There are two hard limits you smash into:

The pivot to horizontal scaling. The industry realized the answer was not a bigger machine but more machines — scale out (horizontal scaling). Run 10, 100, or 10,000 commodity servers, each cheap and replaceable, all serving the same application. This solves the cost curve (commodity hardware is cheap and linear) and the SPOF problem (lose one of 100 servers, lose 1% of capacity, not 100%).

But horizontal scaling created a new problem: If I have 100 identical servers, how does a client know which one to talk to? You cannot hand every user a list of 100 IP addresses. You cannot let clients pick randomly — some servers would be swamped while others idle. You need a single, stable entry point that intelligently spreads traffic.

That entry point is the load balancer. It is the enabling technology of horizontal scaling. Without it, "just add more servers" is impossible. This is why load balancing is concept #1 — it is the foundation on which almost every other concept in this bible is built.

1.3The Problems Load Balancing Solves (Summary)#

2The Recursive Sub-Concept Tree#

Per the strict rule of this bible: before we go deep, every secondary term that appears must be explained from scratch. Here are the foundational sub-concepts you must own before the mechanics make sense.

2.1Sub-Concept: The OSI Model (Layers 4 and 7)#

The OSI model is a 7-layer conceptual framework describing how data moves across a network. For load balancing you only need two layers, but you must understand them precisely.

Mnemonic: Layer 4 knows the envelope (who it's from, who it's to). Layer 7 can open the envelope and read the letter.

2.2Sub-Concept: Virtual IP (VIP)#

A Virtual IP (VIP) is a single IP address that the load balancer advertises to the world as the address of the service. Clients connect to the VIP. Behind the VIP, the LB maintains the list of real server IPs (sometimes called RIPs). The client never sees or knows the real IPs — it thinks it is talking to one machine at the VIP, when in reality it is being fanned out to a whole fleet. The VIP is what makes "one stable entry point for 100 servers" possible.

Why "virtual." An ordinary IP address is tied to one machine's network interface. A VIP is called virtual because it is deliberately not permanently bound to any one machine — it is an address the service owns, which some machine currently holds on the service's behalf and which can be moved to a different machine in seconds without any client noticing. That mobility is the entire point: it is what lets you replace a dead load balancer without a DNS change (§7.2), and it is why the failover mechanism is described in terms of who currently answers ARP for the VIP (§2.8.3) rather than in terms of rewiring anything.

How a machine actually holds a VIP. Concretely, on Linux the LB adds the address to its network interface (ip addr add 1.2.3.4/32 dev eth0) and answers ARP for it. The address now exists in two places: on the interface (so the IP layer accepts packets for it, §0.6) and in every neighbour's ARP cache (so frames get physically delivered to this machine, §2.8.3). Moving the VIP therefore means two operations: remove it from the old machine's interface, add it to the new one's, and broadcast a gratuitous ARP so every peer's cache is corrected immediately rather than after a multi-minute timeout. That is precisely what keepalived automates.

Three ways a VIP is made highly available, each with a different failover speed and cost:

RIP (Real IP) is the counterpart term you will see in LVS documentation: the backends' actual addresses. LVS docs also use DIP (director IP — the LB's own address on the backend-facing side). Knowing this vocabulary is how you read the LVS manual without friction.

2.3Sub-Concept: Upstream / Backend Pool#

The upstream (Nginx terminology) or backend pool / target group (AWS terminology) is simply the named set of real servers the LB distributes traffic to. Servers can be added to or removed from this pool dynamically. A single LB can host many pools (e.g., one pool for /api, another for /static).

2.4Sub-Concept: Health Check#

A health check is a periodic probe the LB sends to each backend server to answer the question "are you alive and ready to serve traffic?" If a server fails its health check, the LB stops sending it traffic (marks it "unhealthy" / "out of rotation"). This is the mechanism that makes server failure invisible to users. Detailed in Section 6.

2.5Sub-Concept: Connection Draining (a.k.a. Graceful Shutdown / Deregistration Delay)#

When you need to take a server out of rotation (to deploy new code, or because it's being scaled down), you don't want to instantly kill it — that would drop the requests currently in flight. Connection draining means: stop sending the server new connections, but allow its existing in-flight requests to complete (up to a timeout, e.g., 30s) before fully removing it. This enables zero-downtime deploys.

2.6Sub-Concept: TLS Termination#

TLS (Transport Layer Security) is the encryption protocol behind HTTPS. TLS termination means the load balancer is the endpoint that decrypts the incoming HTTPS traffic. The client ↔ LB leg is encrypted (HTTPS); the LB ↔ backend leg can then be plain HTTP (inside a trusted private network) or re-encrypted (TLS re-encryption / "TLS passthrough" variants). Terminating TLS at the LB offloads expensive crypto work from your application servers and centralizes certificate management. (More in Section 5.)

That paragraph uses five terms it does not define. Here they are.

What TLS actually does, and why it is expensive. TLS gives three guarantees: confidentiality (an eavesdropper on the wire sees ciphertext), integrity (tampering is detected), and authentication (the client can verify the server really is api.example.com and not an impostor). Establishing those guarantees requires a handshake that runs after the TCP handshake and before any HTTP flows: the client sends a ClientHello listing the cipher suites and TLS versions it supports; the server replies with its choice and its certificate; the two sides perform an asymmetric key exchange (in TLS 1.3, an Elliptic Curve Diffie–Hellman exchange) to agree on a shared secret; and from then on the connection is encrypted with a fast symmetric cipher (typically AES-GCM or ChaCha20-Poly1305) using that secret. The cost is concentrated in the handshake, because asymmetric cryptography is orders of magnitude more expensive per operation than symmetric — a modern core might do a few thousand RSA-2048 handshakes per second but encrypt gigabytes per second symmetrically. That asymmetry is the entire economic argument for terminating TLS at a load balancer: you concentrate the expensive per-connection work in a tier you can scale and accelerate, and you let backends do cheap plaintext.

Round trips matter too. TLS 1.2 costs two extra round trips before the first byte of HTTP; TLS 1.3 cuts it to one, and its 0-RTT resumption mode can send application data in the very first packet on a repeat connection (at the price of replay risk for non-idempotent requests — see File 15). Session resumption (session IDs or session tickets) lets a returning client skip the asymmetric work entirely. This is a load-balancing concern because session tickets must be shared across your LB fleet — if each LB has its own ticket key, a client that lands on a different LB on reconnect suffers a full handshake, and your p99 latency quietly degrades as you add LBs.

A certificate is a file binding a domain name to a public key, signed by a Certificate Authority (CA) the client's operating system or browser already trusts. The client validates the signature chain, checks that the name matches, and checks the expiry. Certificates expire — 90 days for Let's Encrypt, and the industry is shortening this — so automated renewal (ACME) is not a nicety; an expired certificate is a total outage, and it is one of the most common self-inflicted ones in the industry.

SNI (Server Name Indication) is the extension that makes the whole thing work at a load balancer. The problem: the server must present the right certificate, but at handshake time the request's Host: header has not been sent yet — it is inside the encryption that has not been negotiated. SNI solves it by having the client put the hostname **in plaintext in the ClientHello. The LB reads it and selects the matching certificate, which is how one VIP serves hundreds of HTTPS domains. It is also how an L4** load balancer can do a limited kind of hostname routing without decrypting anything: SNI is visible even in a passthrough deployment.

The three deployment modes — this is a question interviewers ask directly:

2.7Sub-Concept: NAT (Network Address Translation)#

NAT is the technique of rewriting the IP addresses (and often ports) inside a packet's headers as it passes through a middlebox, so that the packet appears to come from, or go to, a different machine than it originally did. It exists for two historical reasons, and both matter to load balancing:

Hold this: **DNAT gets the packet to a backend; SNAT gets the reply back through the LB.** NAT-mode load balancing (§3.2) is built from exactly these two operations, and its performance limits are what motivated DSR.

2.8Sub-Concept: MAC Addresses, L2 Segments, ARP, and the Loopback Interface#

These four terms are prerequisites for understanding Direct Server Return (DSR) in §3.2. They live below IP, at Layer 2 (the data-link layer) of the OSI model (§2.1) — the layer where machines on the same local network actually hand frames to each other. Read §0.5 (encapsulation) and §0.6 (the NIC) first if you have not — this section assumes you know what a frame is and how a NIC filters one.

The short version first, then each piece gets its own full treatment in §2.8.1–§2.8.6 below.

Those four bullets are the summary. Now each one properly, because a summary is not an explanation and you cannot debug a DSR deployment from a summary.

2.8.1 The MAC address in full — what it is, where it comes from, what it looks like#

A MAC address (Media Access Control address), also called a hardware address, physical address, or Ethernet address, is a 48-bit (6-byte) identifier belonging to a network interface. It is written as six hexadecimal byte values separated by colons: aa:bb:cc:dd:ee:01. Because it is 48 bits, there are 2⁴⁸ ≈ 281 trillion possible values — enough that every network card ever manufactured can have a globally unique one.

Why it exists at all — the first-principles answer. IP addresses cannot do this job, and understanding why is the point. An IP address is logical and locational: it says "this machine sits in this part of the network topology," and it changes when a machine moves to a different network (your laptop gets a different IP at home than at the office). Routing across the internet requires that locational structure — routers aggregate millions of addresses into a few routing-table entries precisely because addresses are assigned in topological blocks. But at the moment a frame must be physically placed on a wire and delivered to one specific card among the forty plugged into this switch, you need an identifier that is not locational and not reassignable — a name for the hardware itself. That is the MAC address. Layer 3 answers "which network, roughly where in the world"; Layer 2 answers "which physical card, right here."

Structure — the two halves. The 6 bytes split into two 3-byte halves:

Two special bits in the very first byte that you should recognize because they explain the special addresses:

Where you see it. On Linux, ip link show prints each interface's MAC after link/ether. ip neigh show prints the machine's learned mapping of neighbour IPs to MACs (the ARP table, §2.8.3). In a packet capture, tcpdump -e adds the Ethernet header so you can see the source and destination MACs of each frame — without -e, tcpdump hides Layer 2 entirely, which is why so many engineers have never actually looked at one.

The tie-back: the LB's DSR trick is nothing but writing a different value into the 6-byte destination-MAC field of an arriving frame. Everything else in the frame — the entire IP packet, addressed to the VIP — is passed through byte-for-byte unchanged.

2.8.2 Switches, broadcast domains, VLANs, and what "same L2 segment" actually means#

What a switch is and what it actually does. An Ethernet switch is a box with many ports that forwards frames between the machines plugged into it. Its entire intelligence is one data structure: the MAC address table (also called the CAM table, for Content-Addressable Memory, or the forwarding database / FDB), which maps MAC address → the port that address was last seen on. The switch fills this table by itself through backward learning, and the mechanism is worth walking concretely because it explains both DSR and the ARP failure modes.

Take a switch with four ports and three machines, starting with an empty table:

   MAC table (empty)                Port 1: LB          aa:bb:cc:dd:ee:01
   +------+------+                  Port 2: Server A    aa:bb:cc:dd:ee:02
   | MAC  | Port |                  Port 3: Server B    aa:bb:cc:dd:ee:03
   +------+------+                  Port 4: router
   |      |      |
   +------+------+

   after step 1:  | aa:bb:cc:dd:ee:01 | 1 |

   after step 2:  | aa:bb:cc:dd:ee:01 | 1 |
                  | aa:bb:cc:dd:ee:02 | 2 |

Entries age out after a default of about 300 seconds of silence, so that a machine moved to a different port is eventually re-learned. Crucially, the switch never looks at the IP header. It reads 6 bytes of destination MAC and forwards. That is why switching is done in hardware at line rate while routing costs more — and why a MAC-rewriting load balancer (DSR) is dramatically cheaper per packet than an IP-rewriting one (NAT).

Broadcast domain — the precise definition. A broadcast domain is the set of interfaces that will receive a frame sent to ff:ff:ff:ff:ff:ff. A switch floods broadcasts out every port, so everything connected to one switch — or to a set of switches cabled together — is in one broadcast domain. A router does not forward broadcasts; it is where the broadcast domain stops. "Same L2 segment," "same broadcast domain," and "same local network" all name the same thing: machines that can reach each other by MAC address directly, with no router in between. And because ARP (§2.8.3) works by broadcasting, ARP only ever reaches machines in the same broadcast domain — which is the deep reason DSR carries its topology constraint.

VLAN — subdividing one switch into several segments. A VLAN (Virtual LAN) lets one physical switch behave as several independent switches. Each port is tagged with a VLAN ID (a 12-bit number, so 4094 usable VLANs), and the switch will only forward frames between ports in the same VLAN. Frames crossing between switches on a trunk port carry a 4-byte 802.1Q tag inserted into the Ethernet header, holding that VLAN ID. Two consequences: (1) VLANs let one rack of hardware host several isolated broadcast domains — production, staging, and management traffic sharing switches without seeing each other; (2) traffic between two VLANs must go through a router, so two machines on the same switch but different VLANs are not on the same L2 segment and DSR between them will not work. The 4-byte 802.1Q tag also eats into the MTU budget from §0.5, which is why some VLAN deployments quietly need MTU adjustment.

Why cloud VPCs break all of this. An AWS VPC subnet, a GCP subnet, or an Azure VNet looks like a local network — machines share an address range like 10.0.1.0/24 and reach each other in one hop — but underneath there is no shared Ethernet segment at all. Cloud networks are software-defined overlays: your VM's packets are encapsulated by the hypervisor (AWS uses a scheme built on VXLAN-style encapsulation with its own control plane) and tunnelled across the provider's physical fabric. The provider's SDN controller knows the IP-to-host mapping centrally, so it does not need or honour your ARP broadcasts — it answers them synthetically — and it will not deliver a frame you addressed to some other instance's MAC, because the mapping it enforces is IP-based and anti-spoofing checks drop the rest. This is the concrete, mechanical reason "DSR does not work in most cloud VPCs": not a policy decision, but the absence of the L2 broadcast domain the technique requires. It is also why cloud providers give you managed load balancers (NLB/GLB) that solve the response-path problem differently — with their own encapsulation and distributed data planes.

How to check. ip route shows which destinations your machine believes are directly reachable ("on-link", meaning no gateway listed) versus via a router; anything on-link is in your L2 segment. arping <ip> succeeds only within the broadcast domain. bridge fdb show on a Linux bridge, or show mac address-table on a switch, prints the learned MAC table itself.

The tie-back: DSR rewrites only the destination MAC, and a MAC-addressed frame dies at the first router. Therefore the LB and every backend must sit in one broadcast domain. That single sentence contains the entire constraint.

2.8.3 ARP in full — the request/reply lifecycle, the cache, gratuitous ARP, and the failure modes#

The problem ARP solves. Your machine wants to send a packet to 10.0.0.12, and it has determined from its routing table that 10.0.0.12 is on-link — same segment, no router needed. But to put a frame on the wire it needs a destination MAC address, and it has an IP address. Nothing in the IP address reveals the MAC; they are unrelated namespaces. ARP (Address Resolution Protocol) is the lookup that bridges them, and it is deliberately dumb: it asks everyone.

The full lifecycle, with concrete values. Machine 10.0.0.5 (MAC …:01) wants to reach 10.0.0.12:

   ARP REQUEST  (opcode 1)
   sender MAC : aa:bb:cc:dd:ee:01     sender IP : 10.0.0.5
   target MAC : 00:00:00:00:00:00     target IP : 10.0.0.12     <- "who has this?"

   ARP REPLY  (opcode 2)
   sender MAC : aa:bb:cc:dd:ee:02     sender IP : 10.0.0.12    <- "me, and here's my MAC"
   target MAC : aa:bb:cc:dd:ee:01     target IP : 10.0.0.5

The cache and its states. Linux ARP entries move through a small state machine visible in ip neigh show: REACHABLE (confirmed recently, default ~30 seconds of confidence), STALE (past the confidence window but still usable — the kernel will use it and re-validate in the background), DELAY/PROBE (re-validation in progress), and FAILED (nobody answered). The kernel also gets free confirmation from higher layers: if a TCP connection to that neighbour is successfully exchanging data, the ARP entry is refreshed without any ARP traffic. Entry capacity is bounded by net.ipv4.neigh.default.gc_thresh1/2/3 (default 128/512/1024); on a machine with thousands of neighbours — a busy load balancer on a large flat segment — you will hit gc_thresh3, see neighbour table overflow in dmesg, and suffer intermittent packet loss until you raise it. That is a real, specific load-balancer production failure with a real, specific fix.

Gratuitous ARP — the announcement, not a question. A gratuitous ARP is an ARP message a machine sends unprompted, where the target IP equals its own IP. Its purpose is not to ask anything; it is to tell the entire segment "this IP is now at this MAC — update your caches." It comes in two forms (an unsolicited request for one's own IP, which is the common form because requests are broadcast and therefore reach everyone, or an unsolicited reply), and it has two uses, both directly relevant here:

Failure modes you must be able to name.

The tie-back: DSR depends on exactly one machine — the load balancer — answering ARP for the VIP, while the backends silently hold the same address. ARP is the mechanism that must be suppressed for the trick to work, and gratuitous ARP is the mechanism that makes LB failover fast.

2.8.4 The loopback interface in full — and why the VIP goes there#

What it is. The loopback interface (lo on Linux, lo0 on BSD/macOS) is a purely virtual network interface implemented in software by the kernel. It has no cable, no NIC, no MAC address, and no physical existence. Anything transmitted to it is immediately handed back up the receive path of the same machine. Its canonical address is 127.0.0.1 (the whole 127.0.0.0/8 block is reserved for it) plus ::1 in IPv6, and it is always up — it is the one interface that exists even on a machine with no networking at all. Its original purpose is local inter-process communication and testing: your web browser can talk to a server on your own machine at http://127.0.0.1:8080 using the ordinary TCP/IP stack, with no hardware involved.

The three properties that make it the perfect home for a DSR VIP:

**Why a /32 netmask matters.** You add the VIP as 1.2.3.4/32 — a netmask covering exactly one address — rather than, say, /24. A /24 would make the kernel install a route claiming the whole 1.2.3.x network is reachable directly through lo, which would blackhole traffic to every other address in that range: packets destined to 1.2.3.9 would be handed to the local stack instead of out the wire. /32 claims exactly the one address and nothing else. Getting this wrong is a classic self-inflicted outage.

Concrete configuration on a DSR backend (the "loopback VIP config" that the four-bullet summary above waved at):

ip addr add 1.2.3.4/32 dev lo            # accept packets addressed to the VIP
sysctl -w net.ipv4.conf.all.arp_ignore=1 # never answer ARP for it (see §2.8.5)
sysctl -w net.ipv4.conf.all.arp_announce=2

The tie-back: the loopback VIP is what makes a backend say "yes, that IP is me" to a packet the LB hand-delivered by MAC, without that backend ever telling the network it owns the address.

2.8.5 arp_ignore and arp_announce — every value, explained#

These two Linux sysctls are named constantly in DSR documentation and almost never explained. They are the difference between a working DSR deployment and an ARP war. Both exist per-interface under net.ipv4.conf.<iface>. and globally under net.ipv4.conf.all.; **Linux takes the maximum of the all value and the interface value**, which is why setting all is the safe move.

**arp_ignore — when will this machine answer an ARP request?**

**arp_announce — what source IP will this machine put inside the ARP requests it sends?**

The subtle failure this prevents: when the backend initiates its own outbound traffic and needs to ARP for the gateway, the kernel might, with default settings, use the VIP as the sender IP inside that request — thereby announcing "the VIP is at my MAC" as a side effect of an ordinary lookup. Peers cache it, and you get the same storm through the back door.

How to verify it worked: from another machine on the segment, run arping -D <VIP> and confirm exactly one responder — the LB — replies. Or run tcpdump -e -n arp on the segment and watch: you should see requests for the VIP answered only by the LB's MAC. If two MACs answer, you have found your bug.

Persisting it. sysctl -w is lost on reboot; production configuration goes in /etc/sysctl.d/99-dsr.conf. A backend that reboots and comes back with arp_ignore=0 will silently poison the segment — a genuinely nasty incident, because the fleet works fine until the one box that rebooted starts stealing traffic.

The tie-back: these two settings are the entire enforcement mechanism for "only the LB advertises the VIP." Everything else in DSR is just packet plumbing.

2.8.6 Putting it together — one DSR packet, traced through every layer#

Setup: LB at 10.0.0.5 (MAC …:01), backend at 10.0.0.12 (MAC …:02), VIP 1.2.3.4 configured on the LB's eth0 and on the backend's lo, client at 203.0.113.9, all on one L2 segment behind router 10.0.0.1 (MAC …:FF).

Total load-balancer involvement: it touched 6 bytes, on the inbound direction only, and never saw the response at all. That is the whole economic argument for DSR.

2.9Sub-Concept: Deployment Strategies — Rolling, Blue-Green, Canary#

Load balancers are the instrument of safe deployments, and these three terms appear throughout this file (and in every system design interview), so define them properly:

Decision rule: rolling is the cheap default; blue-green when you need instant rollback and can afford 2× capacity for an hour; canary when the change is risky enough that you want real-traffic proof before betting the fleet on it. Serious shops combine them: canary first, then roll.

With these sub-concepts defined, we can now go under the hood.

3Deep-Dive Mechanics#

3.1The Life of a Request Through a Load Balancer#

Let's trace an HTTPS request end-to-end through a Layer 7 load balancer. This is the exact flow you should be able to draw on a whiteboard.

   CLIENT                    LOAD BALANCER (VIP: 1.2.3.4:443)            BACKEND POOL
   (browser)                                                          (10.0.0.11..13:8080)
      |                                                                       |
      |  1. DNS lookup: api.example.com -> 1.2.3.4                            |
      |----------------------------------------------------------------------|
      |  2. TCP handshake (SYN, SYN-ACK, ACK) to 1.2.3.4:443                  |
      |------------------------------------------->|                          |
      |  3. TLS handshake; LB presents certificate |                          |
      |<------------------------------------------>|  (TLS terminated here)   |
      |  4. HTTP request:  GET /api/users HTTP/1.1 |                          |
      |      Host: api.example.com                 |                          |
      |------------------------------------------->|                          |
      |                                            | 5. LB reads L7 data:     |
      |                                            |    path=/api/users       |
      |                                            |    picks pool + algorithm|
      |                                            |    picks server 10.0.0.12|
      |                                            | 6. LB opens/reuses conn  |
      |                                            |---------------------->|   |
      |                                            |    (plain HTTP or mTLS)  | 7. app processes
      |                                            |<----------------------|   |    returns 200
      |  8. LB relays response back to client      |                          |
      |<-------------------------------------------|                          |

Key mechanical points:

3.2The Four Forwarding Architectures: Full Proxy, NAT, DSR, and IP Tunneling#

This is a classic senior-level distinction. There are four fundamentally different ways an LB can move a packet to a backend, and they differ in exactly one dimension that matters enormously at scale: what does the LB rewrite, and which direction(s) of traffic must flow through it? (These map directly to the modes of LVS — Linux Virtual Server, the kernel L4 balancer, which calls them NAT mode, DR mode, and TUN mode; full proxy is what Nginx/HAProxy/ALB do.)

(A) Full Proxy Mode (the default, e.g., Nginx, HAProxy, AWS ALB): Both request and response flow through the LB. The LB terminates the client's TCP connection entirely and originates a brand-new connection to the backend — two independent conversations glued together by the LB. Response traffic returns via the LB.

(B) NAT Mode (L4 — the "rewrite the addresses" technique): The LB does not terminate the connection; it forwards the client's own packets, rewriting their headers using the NAT operations defined in §2.7. This is the classic LVS-NAT mode and the mental model behind "an L4 LB uses NAT" (§5.1).

(C) Direct Server Return (DSR) / Direct Routing (L4 technique): The request goes client → LB → backend, but the response goes backend → client directly, bypassing the LB entirely. The LB only sees inbound traffic. (Uses the Layer-2 machinery from §2.8 — read that first.)

(D) IP Tunneling Mode (LVS-TUN — "DSR across routers"): Same goal as DSR (responses bypass the LB) but without the same-L2-segment constraint.

Comparison — the one table to remember:

Decision rule: full proxy for anything needing HTTP intelligence (the web-app default); NAT mode for simple L4 balancing where response volume is modest; DSR/tunneling when responses dwarf requests (video, downloads, CDN edges) and you can pay the network-config complexity.

3.3How the LB Maintains State#

A Layer 4 LB maintains a connection tracking table (conntrack). For every active TCP flow it stores a 5-tuple:

(source IP, source port, dest IP, dest port, protocol) -> chosen backend server

This ensures every packet of the same TCP connection goes to the same backend (you cannot send packet 1 of a TCP stream to server A and packet 2 to server B — TCP state would break). This table is why L4 LBs must be stateful per-connection, and why LB failover is tricky (Section 11).

A Layer 7 LB additionally parses the HTTP stream, so it can make a new routing decision per HTTP request, even multiplexing many requests from one client connection across different backends (especially with HTTP/2 and keep-alive).

4Routing Algorithms — Exhaustive Breakdown#

The "policy" for choosing which backend gets the next request is the routing (or balancing) algorithm. Interviewers love asking you to compare these. Here is every one that matters.

4.1Round Robin#

Mechanism: Maintain a pointer. Server 1, then 2, then 3, ..., then N, then back to 1. Purely sequential rotation.

4.2Weighted Round Robin#

Mechanism: Assign each server a weight (e.g., a beefy 32-core box gets weight 4, an old 8-core box gets weight 1). The box with weight 4 receives 4× the requests per cycle.

4.3Least Connections#

Mechanism: Route the new request to the server with the fewest active connections right now.

4.4Weighted Least Connections#

Combines the above two: normalize active connections by server weight/capacity. Route to whoever has the lowest active_connections / weight ratio. Best general-purpose adaptive algorithm for mixed fleets.

4.5Least Response Time (Least Latency)#

Mechanism: Route to the server with the lowest combination of active connections and lowest measured recent response time (EWMA — exponentially weighted moving average of latency).

4.6IP Hash / Source Hash#

Mechanism: hash(client_source_IP) mod N → picks the server. The same client IP always maps to the same server (as long as N doesn't change).

4.7Consistent Hashing (preview)#

Mechanism: Instead of hash mod N, map both servers and requests onto a hash ring, and route each request to the next server clockwise on the ring. Adding/removing a server only remaps 1/N of keys instead of nearly all of them.

4.8Random (with Two Choices — "Power of Two Choices")#

Mechanism: Pick two servers at random, then send the request to whichever of the two has fewer connections.

4.9Summary Table#

5Layer 4 vs. Layer 7 — The Core Distinction#

This is the single most common load-balancing interview question. Master it cold.

5.1Layer 4 Load Balancer#

Operates on TCP/UDP. Makes routing decisions using only IP + port. Does not decrypt TLS or read HTTP.

How it works: Uses one of the L4 forwarding modes from §3.2 — NAT (rewrite the packet's destination/source IPs, §2.7), DSR (rewrite only the MAC, responses bypass the LB), or IP tunneling (encapsulate and forward). In every case it merely rewrites headers and forwards the client's own packets — it never terminates the connection or parses the payload. Because it does almost no work per packet, it is blazingly fast and handles enormous throughput with tiny latency (microseconds).

Examples: AWS Network Load Balancer (NLB), Google TCP/UDP LB, LVS (Linux Virtual Server), F5 in L4 mode.

Characteristics:

5.2Layer 7 Load Balancer#

Operates on HTTP/HTTPS/gRPC. Terminates the connection, reads the full application message, and routes on content.

Examples: AWS Application Load Balancer (ALB), Nginx, HAProxy (in HTTP mode), Envoy, Traefik, Google HTTPS LB.

Characteristics:

5.3The Decision Rule#

Use Layer 7 when you need to make decisions based on what the request is (URLs, headers, cookies) or need HTTP features (TLS termination, path routing, WAF). This is the default for web applications and APIs.

Use Layer 4 when you need raw speed and protocol independence, when you're balancing non-HTTP traffic (databases, custom protocols, gaming), or when you want to preserve the client's source IP and do minimal processing.

Real production pattern: They are frequently stacked. An L4 NLB at the very edge (absorbs huge connection volume, handles DDoS, static IPs), forwarding to a fleet of L7 proxies (Envoy/ALB) that do the smart HTTP routing. Best of both worlds.

6Health Checks, Failover & Session Persistence#

6.1Health Checks — Deep Dive#

A health check answers "should this server receive traffic?" There are two philosophies:

(A) Active health checks (LB → server): The LB proactively pings each backend on a schedule.

Tunable parameters (know these terms):

(B) Passive health checks (observe real traffic): Instead of (or in addition to) probing, the LB watches real request outcomes. If a server returns a burst of 5xx errors or connection resets, eject it. Envoy calls this outlier detection. More responsive than active checks (catches problems between probe intervals) but reactive (some real users hit the error first).

6.2Liveness vs. Readiness (crucial distinction)#

6.3Session Persistence (Stickiness) — Deep Dive#

The problem: some backends store per-user state in memory (e.g., a shopping cart, a WebSocket session, a login session cached locally). If request 1 lands on server A (which now holds your cart) but request 2 lands on server B (which knows nothing), the app breaks.

Solutions, worst to best:

7Pros, Cons & Trade-offs#

7.1Benefits (Why you always have one)#

7.2Costs & Trade-offs (What interviewers probe)#

8Interview Diagnostic Framework (Symptom → Solution)#

Use this "if you see this, reach for load balancing (and which flavor)" map.

9Real Implementations & Product Landscape#

Everything so far has been mechanism. This section is the catalogue: the actual software and services that implement these mechanisms, what each one is, what it is genuinely good at, where it falls down, and — the part that matters in an interview and on the job — which one you should pick in which situation. When this file (or an interviewer) says "use NGINX" or "put an ALB in front," this is the section that makes that sentence mean something. A name without its landscape teaches nothing: ALB and NLB are both "AWS load balancers" and they are not remotely interchangeable.

The landscape divides into six families, ordered roughly from lowest layer to highest.

9.1Family 1 — Kernel-space and hardware-accelerated L4 balancers#

These live inside the operating system kernel or on the NIC itself. They do not terminate connections, do not understand HTTP, and are optimized for one thing: moving enormous packet volumes per box with minimal CPU. This is the family that uses the NAT / DSR / tunnel modes of §3.2 directly.

LVS / IPVS (Linux Virtual Server). LVS is the original open-source L4 load balancer, built into the Linux kernel since 2.4 as the IPVS module, created by Wensong Zhang in 1998. It is not a daemon that receives and re-sends traffic; it is a hook inside the kernel's packet-processing path, which is why it is fast — a packet is switched to a backend without ever being copied to user space or touched by a scheduler. It is the canonical implementation of the three forwarding modes in §3.2 (NAT, DR, TUN) — those mode names come from LVS — and it implements about ten scheduling algorithms including round robin, weighted round robin, least connection, weighted least connection, and source hashing (§4). You configure it with ipvsadm, and in practice you never run it alone: it is paired with keepalived, which adds health checking of the backends and VRRP-based failover of the VIP between two LVS boxes (§7.2). Best at: extremely high-throughput L4 balancing on hardware you own, especially with DR mode where a single mid-range box can front hundreds of gigabits of egress. Limits: no L7 whatsoever — no TLS termination, no path routing, no header manipulation, no retries; configuration is imperative and awkward to automate; DR mode drags in the whole L2 constraint of §2.8. Cost: free, but you are operating Linux boxes and a VRRP pair yourself. Reach for it when: you run on-prem or bare metal, response volume dwarfs request volume, and you need a cheap L4 tier under your L7 tier. Not when: you are in a cloud VPC (DR mode won't work, §2.8.2) or you need anything HTTP-aware.

Google Maglev. Maglev is Google's software network load balancer, described in their 2016 NSDI paper, and it is the design that changed how the industry thinks about L4 tiers. Instead of a special appliance, Maglev runs as a user-space program on ordinary commodity servers, bypassing the kernel networking stack entirely (it takes packets directly from the NIC via a kernel bypass and a shared memory ring). Many Maglev machines simultaneously announce the same VIP to the network via BGP, and the routers spread incoming flows across them using ECMP (Equal-Cost Multi-Path) — hardware-level load balancing of the load balancers. Two design details are famous. First, it forwards using GRE encapsulation and DSR-style return, so responses bypass Maglev entirely. Second, and more importantly, it uses a purpose-built consistent hashing scheme (Maglev hashing) that gives near-perfect load distribution and ensures that different Maglev machines, having never spoken to each other, independently choose the same backend for the same flow — which is what makes it safe for a router to reshuffle a flow to a different Maglev box mid-connection. Best at: planetary-scale L4 with no single-machine ceiling; adding capacity is adding a machine. Limits: it is Google-internal; you consume it as GCP's Network Load Balancer, not as software you install. The interview value: naming Maglev and explaining ECMP + consistent hashing is the difference between "run two load balancers" and "here is how you scale the load-balancing tier itself."

Facebook Katran and Cloudflare Unimog are the modern open/published equivalents, both built on XDP (eXpress Data Path) and eBPF — a Linux facility that lets you attach a small verified program to the NIC driver's receive path so packets are inspected and redirected before the kernel allocates its usual per-packet data structures. This gets you Maglev-class performance (tens of millions of packets per second per box) on stock Linux. Katran is open source and usable; both use consistent hashing and encapsulation for the same reasons Maglev does. Reach for these when you are building your own L4 tier at very large scale and cannot use a cloud provider's.

Hardware appliances — F5 BIG-IP, Citrix ADC (formerly NetScaler), A10. These are dedicated physical boxes with purpose-built ASICs, sold with support contracts, and they still run a large fraction of enterprise and financial-sector traffic. They do L4 and full L7, plus a long tail of enterprise features (WAF, SSL VPN, sophisticated iRules scripting). Best at: regulated environments needing a vendor to call, hardware crypto acceleration, and feature depth accumulated over twenty years. Limits: very expensive (list prices routinely in the tens of thousands per appliance plus annual support), long procurement cycles, capacity bought in fixed increments, and a configuration model that resists modern GitOps automation. Reach for them when you are in an enterprise that already owns them or has compliance requirements naming them. Not when you are building anything cloud-native — the economics and the operational model both fight you.

9.2Family 2 — User-space software proxies (the L7 workhorses)#

These are full proxies (§3.2 A): they terminate the client's connection, parse HTTP, and open their own connection to a backend. This is the family you will actually configure in most jobs.

NGINX. Written by Igor Sysoev around 2004 to solve the C10K problem (serving ten thousand simultaneous connections on one machine, which the then-standard thread-per-connection model could not do), NGINX's defining architectural choice is an event-driven, asynchronous, single-threaded-per-worker model: a small fixed number of worker processes, each running an event loop over thousands of connections, rather than a thread or process per connection. That is why its memory footprint per connection is a few kilobytes instead of a megabyte-scale thread stack, and why it handles slow clients gracefully. It is simultaneously a web server, a reverse proxy, a load balancer, and a cache — which is its great practical advantage: one binary does static file serving, TLS termination, and balancing. It supports round robin, weighted round robin, least connections, IP hash, and generic hash including consistent hashing (§4), plus passive health checks. Limits worth knowing: in the free open-source version, active health checks (§6.1) and dynamic reconfiguration of the upstream pool via an API are NGINX Plus (commercial) features; open-source NGINX picks up backend changes by reloading its configuration file, which is graceful but is still a config-file-driven workflow rather than an API-driven one. Reach for it when: you want one well-understood, extremely stable component that terminates TLS, serves static assets, caches, and balances — the default choice for a single-service or small-fleet deployment. Not when: you need per-request dynamic backend discovery in a rapidly changing service mesh (Envoy fits better) or deep TCP-level observability (HAProxy fits better).

HAProxy. Written by Willy Tarreau starting in 2000, HAProxy is the specialist: it is only a proxy and load balancer, and that focus shows. It has the richest health-checking system of the open-source options (including expect-based checks that assert on response content, agent checks where the backend reports its own weight, and slow-start ramping for newly-added servers), the best-in-class runtime statistics page and CSV/Prometheus metrics, a runtime admin socket that lets you change weights, drain servers, and add backends without a reload, and unusually precise timeout controls. It operates in both TCP mode (L4) and HTTP mode (L7). Best at: operational control and observability; if you need to know exactly why a request was slow, HAProxy's logs tell you the queue time, the connect time, the server response time, and the total time as separate fields. Limits: it does not serve static files or act as a general web server, so you usually run it alongside something that does; its configuration language, while powerful, is dense. Reach for it when: load balancing is the whole job and you care about health-checking sophistication, draining, and metrics — a high-traffic API tier, a database read-replica pool, a TCP service. Not when: you also wanted a web server and a cache in the same process.

Envoy. Created at Lyft in 2016 and now a CNCF project, Envoy was designed for a world NGINX and HAProxy predate: thousands of ephemeral microservice instances that appear and disappear continuously. Its defining choice is the xDS APIs — a set of gRPC/REST discovery services (LDS for listeners, RDS for routes, CDS for clusters, EDS for endpoints) through which a control plane pushes configuration to Envoy dynamically, at runtime, with no reload at all. Configuration is data streamed from a controller, not a file on disk. On top of that it has native HTTP/2 and gRPC support (including gRPC-specific load balancing, which matters because gRPC multiplexes many requests over one long-lived connection and therefore defeats connection-level balancing), first-class distributed tracing, per-upstream circuit breaking and outlier detection (automatic ejection of misbehaving hosts), retries with budgets, and the panic threshold behaviour described in §11.2. It is the data plane inside Istio, Consul Connect, and AWS App Mesh, and it is what AWS's own ALB-adjacent tooling and many CDNs build on. Limits: it is heavy — meaningfully more memory and CPU per instance than NGINX — and its configuration is verbose YAML/protobuf that is genuinely painful to hand-write; it presumes a control plane, and if you don't have one you are doing the hardest part yourself. Reach for it when: you are running a service mesh (File 10), you need gRPC-aware balancing, or your backend set changes constantly. Not when: you have three static backends and a config file would have taken five minutes.

Traefik. A Go proxy designed for container platforms, whose distinguishing feature is automatic configuration by service discovery: it watches the Docker, Kubernetes, or Consul API and materializes routes from labels/annotations on your services, so deploying a container with the right label makes it routable with no configuration file edited anywhere. It also automates TLS certificates via ACME/Let's Encrypt out of the box. Best at: developer ergonomics on container platforms — the fastest path from "container running" to "HTTPS URL working." Limits: less battle-tested at extreme scale than NGINX/HAProxy, and historically less predictable performance under heavy load. Reach for it when: you run Docker Compose or a small Kubernetes cluster and want minimal configuration ceremony. Not when: you are at a scale where you need to reason hard about tail latency.

Caddy occupies a similar niche with the strongest automatic-HTTPS story of any server (certificates are issued and renewed with zero configuration); **Apache httpd with mod_proxy_balancer still exists and works, but its process/thread model makes it a poor fit for high-concurrency proxying and there is no reason to choose it for new work; Varnish** is a caching reverse proxy that does balance across backends but whose real purpose is HTTP caching (File 02) — do not select it as a load balancer, select it as a cache that happens to have one.

9.3Family 3 — Managed cloud load balancers#

Here you buy the operation, not the software. The critical thing to internalize is that cloud providers sell several distinct products at different layers, and picking the wrong one is a common and expensive mistake.

AWS Application Load Balancer (ALB) — Layer 7 only. Terminates TCP and TLS, parses HTTP/HTTPS/HTTP-2/gRPC/WebSocket, and routes on host header, URL path, HTTP method, query string, source IP, and arbitrary headers. It integrates with AWS Certificate Manager (free managed TLS certificates with automatic renewal), AWS WAF (web application firewall), Cognito (authentication at the edge), and can target EC2 instances, IP addresses, Lambda functions, and ECS/EKS containers. It supports weighted target groups, which is how you do canary and blue-green deploys (§2.9) natively. It performs active HTTP health checks. Its properties follow from being a full proxy: it cannot preserve the client's source IP at the TCP layer (the backend sees the ALB's IP; the real client IP arrives in the X-Forwarded-For header, §11.1), it adds a millisecond or two of latency, its throughput scales elastically but not instantly (very sharp spikes require pre-warming or a design that tolerates a scaling lag), and it only speaks HTTP-family protocols. Pricing model: an hourly charge plus LCU (Load Balancer Capacity Units), a composite metric billing on whichever of new connections/sec, active connections, processed bytes, or rule evaluations is highest — meaning cost tracks your traffic shape, not just its volume.

AWS Network Load Balancer (NLB) — Layer 4 only. Operates on TCP, UDP, and TLS at the connection level. It is built on AWS's Hyperplane distributed data plane and delivers ultra-low latency (tens of microseconds of added latency), millions of requests per second, and instant scaling with no pre-warming. It can give you a static IP per availability zone (and supports Elastic IPs), which is exactly what you need when a customer must whitelist your address in their firewall. Critically, it preserves the client source IP for instance targets — the backend genuinely sees the client's address — because it does not proxy in the ALB sense. It can also terminate TLS if you want that at L4. Limits, all following from being L4: no path or header routing, no X-Forwarded-For insertion (use Proxy Protocol v2 if you need client info with IP targets), no HTTP-level retries, no WAF integration, no request-level metrics. Pricing: hourly plus NLCUs, and generally cheaper than ALB at high byte volumes.

AWS Gateway Load Balancer (GWLB) — the one people forget. It is not for balancing your application at all; it exists to insert third-party virtual network appliances (firewalls, IDS/IPS, deep packet inspection) transparently into your traffic path. It operates at Layer 3, uses the GENEVE encapsulation protocol on port 6081 to hand packets to appliance fleets while preserving the original packet, and keeps flows pinned to one appliance instance. Reach for it when a security team needs all VPC traffic to traverse an inspection fleet, and you need that fleet to be horizontally scalable and health-checked. Mentioning GWLB when an interviewer asks "what AWS load balancers exist" is a strong signal you know the platform rather than the two famous names.

AWS Classic Load Balancer (CLB) is the deprecated original, spanning a limited L4/L7 hybrid. It exists only in old accounts. Do not choose it for new work; know it exists so you recognize it in a legacy diagram.

GCP Cloud Load Balancing is architected differently from AWS's in a way worth knowing: its Global External HTTP(S) Load Balancer is a single anycast IP advertised from Google's edge worldwide, so a user in Tokyo and a user in Frankfurt hit the same address and are absorbed at their nearest edge, then carried to the backend region over Google's private backbone — no DNS-based geo-routing, no per-region addresses, and failover between regions with no DNS change. AWS's equivalent capability requires composing CloudFront and/or Global Accelerator with regional ALBs. GCP's Network Load Balancer is the passthrough L4 product built on Maglev (§9.1), and it uses DSR-style return, which is why it preserves client IP and adds almost no latency.

Azure splits the same way: Azure Load Balancer (L4, regional), Application Gateway (L7, regional, with an integrated WAF), and Azure Front Door (L7, global, anycast, CDN-integrated).

9.4Family 4 — Edge / CDN / global traffic management#

Cloudflare, Fastly, Akamai, AWS CloudFront + Global Accelerator. These are load balancers whose distinguishing property is where they run: hundreds of points of presence worldwide, absorbing traffic within milliseconds of the user. They provide anycast VIPs, DDoS absorption at multi-terabit scale (a capability no single-region LB can offer, because absorbing a volumetric attack requires more ingress capacity than the attack has egress), TLS termination at the edge, and origin load balancing with health checks and geo-failover. Fastly's differentiator is instant (sub-second) cache purging and VCL-based edge programmability; Cloudflare's is the breadth of its free tier and Workers (edge compute); Akamai's is the largest and most deeply embedded PoP footprint. Reach for these when your users are global, or when you need DDoS protection, or when static/media content dominates your response bytes. They are covered fully in File 05 (CDNs) — the load-balancing angle is simply that geographic routing is load balancing with distance as a cost function.

DNS-based global load balancing (Route 53 latency/geolocation/weighted routing, NS1, Akamai GTM) deserves a distinct mention because it is a load balancer that owns no data path at all: it makes its decision at name-resolution time by handing different clients different IP addresses. Benefit: free, infinitely scalable, works for any protocol. Cost: DNS caching means failover is bounded by TTL and by resolvers that ignore TTLs, so it is a coarse instrument — minutes, not milliseconds — and it balances resolvers, not users, so a single large ISP resolver can steer a whole population at once. Mitigation: short TTLs plus health-checked records, or move the job to anycast, which fails over at BGP speed rather than DNS speed.

9.5Family 5 — Kubernetes and service-mesh load balancing#

In Kubernetes there are four distinct load-balancing layers, and conflating them is one of the most common sources of confusion:

Client-side load balancing belongs here too: with gRPC's built-in balancing, or the older Netflix Ribbon, the client holds the list of backends (from a registry like Eureka or Consul) and picks one itself, with no middlebox at all. Benefit: one fewer network hop, no LB tier to scale, and it solves gRPC's connection-multiplexing problem naturally. Cost: every client must embed the logic (hard in a polyglot fleet), and rolling out a balancing change means redeploying every client. Mitigation: the sidecar model — put the logic in a proxy next to the client instead of inside it, which is exactly the argument for a service mesh.

9.6The comparison table#

9.7Decision rules — say these out loud in an interview#

9.8Anti-recommendations — the common wrong picks#

10Real-World Engineering Scenarios#

10.1Netflix — Zuul + Eureka (L7, self-healing edge)#

Netflix's edge gateway is Zuul, a Layer 7 load balancer/router they open-sourced. End-to-end flow:

10.2Google — Maglev (software L4 at planet scale)#

Google's Maglev is a software Layer 4 load balancer running on commodity servers (published paper, 2016).

10.3AWS — ALB + NLB Stack for a SaaS App#

A typical production topology on AWS:

11Interview Gotchas & Failure Modes#

11.1Classic Interviewer Questions (and the crisp answers)#

11.2Failure Modes & Mitigations#

12Whiteboard Cheat Sheet#

                         ┌────────────────────────┐
   Clients  ───DNS──►    │  Redundant LB pair/VIP  │   (never a single box!)
                         │  ┌──────────────────┐   │
                         │  │ L4 (NLB) edge     │   │  fast, TCP/UDP, static IP, DDoS
                         │  └───────┬──────────┘   │
                         │          ▼               │
                         │  ┌──────────────────┐   │
                         │  │ L7 (ALB/Envoy)    │   │  TLS term, path/host routing,
                         │  └───────┬──────────┘   │  retries, WAF, metrics
                         └──────────┼──────────────┘
              health checks ┌───────┼───────┬───────────┐
              (/healthz)    ▼       ▼       ▼           ▼
                        [srv1]   [srv2]   [srv3]  ...  [srvN]   ◄─ stateless!
                                                                    session in Redis
Algorithm picks:  RR / weighted-RR / least-conn / least-latency / P2C / consistent-hash
Deploy safely:    drain → deploy → healthcheck → re-add   (rolling / blue-green / canary)
Failure guards:   panic mode, retry budgets, circuit breakers, headroom, autoscale

The one-sentence summary to give an interviewer:

"A load balancer is the entry point that makes horizontal scaling and high availability possible; choose L4 for raw speed and non-HTTP protocols and L7 for content-aware routing and HTTP features, run it redundantly so it's not a SPOF, keep your servers stateless so you don't need stickiness, and defend against cascading failure with health checks, panic mode, retry budgets, and circuit breakers."

End of File 01. Next up → File 02: Caching Strategies (write-through, write-back, cache-aside, eviction policies, and the thundering-herd problem). Prompt me to continue.