Concept 10: Microservices & Service Mesh

System Design Bible — File 10 of 10 — THE CAPSTONE Difficulty: ★★★★★ (Most Advanced Architecture) Prerequisites: Everything (Files 01–09). The framing you need is distilled in Section 0.

Table of Contents#

0Prerequisites — What You Must Understand First#

This capstone assumes you've absorbed Files 01–09. Beyond that, three framing ideas make the whole architecture click.

0.1Monolith vs microservices — what actually differs#

A monolith is an application built as one single deployable unit: all the features (users, payments, inventory, notifications) live in one codebase, compile into one artifact, run in one process, and typically share one database. To change anything, you rebuild and redeploy the whole thing. Microservices split that same application into many small, independently-deployable services, each owning one business capability and — critically — its own private database that others can't touch directly. To change the payments service, you redeploy only payments, and no other team is affected. The difference isn't "small files vs big files"; it's about the unit of deployment and ownership: a monolith is one big unit everyone shares, microservices are many small units teams own separately. Get this crisp, because every benefit (independent deploys, independent scaling, fault isolation) and every cost (network calls, distributed data, operational sprawl) flows directly from this one structural choice.

0.2The most important sentence: a function call becomes a network call#

Here is the mental model that explains all the complexity in this file. In a monolith, when the checkout code needs to charge a card, it makes a function call — paymentModule.charge(order) — which runs in the same process, in the same memory, on the same machine. It's effectively instant (nanoseconds), it always either returns or crashes the whole program (no in-between), and there's no network involved. When you split payments into its own microservice, that exact same operation becomes a network call — an HTTP/gRPC request across the wire to another machine. And a network call is a completely different beast: it takes milliseconds (thousands of times slower), it can hang, time out, or fail while your service is otherwise fine (the "partial failure" of File 08 §0.2 — the payment service might be up, down, or just slow, and you can't tell which), and the network itself is unreliable. So microservices don't remove complexity — they convert simple, reliable, in-process function calls into slow, unreliable, distributed network calls. Every pattern in this file — service discovery, circuit breakers, retries, timeouts, sagas, the service mesh — exists to cope with the consequences of that one conversion. If you remember nothing else entering this file: the network call is the whole problem.

0.3Conway's Law — why microservices are an organizational choice, not just a technical one#

Conway's Law (1967) observes that "organizations design systems that mirror their own communication structure." If you have one big team, you tend to build one big monolith; if you have twenty small autonomous teams, a monolith forces them all to coordinate on one codebase and one deploy pipeline — which becomes an unbearable bottleneck (merge conflicts, release trains, one team's bug blocking everyone's release). Microservices align software boundaries with team boundaries: each small team owns a service end-to-end (builds it, deploys it, runs it, is on-call for it — "you build it, you run it"), and can move at its own pace without coordinating with everyone else. This reframing is essential and easy to miss: **microservices are primarily a solution to an organizational scaling problem** (letting many teams move independently), and the technical benefits (independent scaling, fault isolation, polyglot tech) follow from that. It's why the correct answer to "should we use microservices?" depends as much on your team/org size as on your traffic — and why premature microservices (a tiny team splitting into 30 services) is a classic, costly mistake. Hold it: the architecture mirrors the org chart, on purpose.

1Architectural Definition & "The Why"#

1.1The Plain Definition#

A microservices architecture structures an application as a collection of small, independently deployable services, each owning a single business capability (e.g., payments, inventory, user-profile, notifications), each with its own database, communicating over the network (HTTP/gRPC/File 07 messaging). This contrasts with a monolith — one large codebase, one deployment, one shared database.

A service mesh is a dedicated infrastructure layer that handles service-to-service communication (discovery, load balancing, retries, security, observability) transparently, by intercepting traffic with a sidecar proxy deployed next to each service — so the services themselves don't contain networking/resilience logic.

1.2"The Why" — The Monolith Scaling Wall (Organizational, Not Just Technical)#

The monolith is the right starting point (simple, one deploy, easy transactions). But as the codebase and the team grow, monoliths hit walls that are as much organizational as technical:

Conway's Law (the deep "why"): "Organizations design systems that mirror their communication structure." Microservices align software boundaries with team boundaries — each small team owns a service end-to-end (build it, run it, deploy it independently). Microservices are fundamentally a way to let many teams move independently and fast. The technical benefits (independent scaling, fault isolation, polyglot) follow from that.

What microservices solve: independent deploys (team autonomy + velocity), independent scaling (scale only the hot service — File 01), fault isolation (one service down ≠ all down), polyglot freedom, and clear ownership.

The brutal catch: you trade in-process function calls for network calls, which means you inherit every hard problem in Files 01–09 at once — load balancing, caching, discovery, partial failure, distributed data, consistency, messaging. A monolith's method call becomes a distributed-systems problem. Microservices don't remove complexity; they move it from the code into the network and operations. That relocation is why you need service discovery, circuit breakers, sagas, and service meshes — this whole file.

2The Recursive Sub-Concept Tree#

2.1Sub-Concept: Service (Bounded Context)#

Each microservice owns a bounded context (a Domain-Driven Design term): a well-defined slice of the business domain with its own model and language. Good boundaries = high cohesion inside, loose coupling outside. Bad boundaries are the #1 cause of microservice failure — split wrong and you get a "distributed monolith" (services so chatty and coupled they must deploy together, giving you all the pain of both worlds).

2.2Sub-Concept: Database-per-Service#

Each service owns its data privately — other services cannot touch its database directly; they must go through its API. This is what enables independent evolution (a service can change its schema without breaking others) and true decoupling. But it shatters the single database — no cross-service joins, no cross-service ACID transactions (this is why you need Sagas, Section 5, and it's File 04's cross-shard problem generalized to services).

2.3Sub-Concept: API Gateway#

Sub-Concept: API Gateway. A single entry point that sits in front of all the microservices (a specialized Layer 7 load balancer, File 01). Clients call the gateway, not services directly. It handles cross-cutting concerns in one place: authentication/authorization, rate limiting (File 03), TLS termination, request routing to the right service, response aggregation (fan out to several services and combine their answers into one response, so a mobile client makes 1 call instead of 6), caching (File 02), and observability. A related refinement is the ***Backend-for-Frontend (BFF)* pattern: instead of one gateway serving every client type, you run a separate, tailored gateway per client — one BFF for the mobile app (small payloads, aggressive aggregation), one for the web app, one for partner APIs — each owned by the team that owns that frontend, so gateway logic doesn't become a shared bottleneck fought over by every client team. Without it, every client would need to know every service's address and re-implement auth/limiting. The gateway is the north-south (client↔system) traffic manager; the service mesh (Section 6) is the east-west** (service↔service) manager.

2.4Sub-Concept: East-West vs North-South Traffic#

2.5Sub-Concept: Partial Failure#

The defining reality of distributed systems: in a monolith, a function call either returns or the whole process crashes. In microservices, a call to another service can hang, time out, return an error, or return slowly while your service is otherwise fine. Some services are up, some are down, some are slow — partial failure. Nearly every resilience pattern in this file exists to handle partial failure gracefully instead of letting it cascade into total failure.

2.6Sub-Concept: Cascading Failure#

The nightmare scenario: service D slows down → service C's calls to D pile up (threads blocked waiting) → C exhausts its threads/connections and slows → C's callers (B) pile up → ... → the entire system collapses because one leaf service degraded. A single slow dependency drags down everything upstream. Circuit breakers, timeouts, and bulkheads (Section 4) exist specifically to stop this chain reaction.

3Service Discovery#

3.1The Problem#

In the cloud, service instances are ephemeral: autoscaling adds/removes them, deploys replace them, failures kill them — their IP addresses change constantly. So "service C needs to call service D" can't use a hardcoded IP. How does C find a healthy, current instance of D? That's service discovery. (This is File 01's "which backend?" problem, now inside the system and dynamic.)

3.2The Service Registry#

Sub-Concept: Service Registry. A database of currently-available service instances — a live "phone book" mapping service name → healthy instance addresses. Instances register on startup ("I'm payments, at 10.2.3.4:8080, and I'm healthy") and are deregistered on shutdown/failure. The registry continuously health-checks (File 01) entries and evicts dead ones. Discovery = querying this registry. It must be highly available and reasonably consistent — so it's typically backed by a consensus store (File 08): Consul, etcd, ZooKeeper, or Eureka. (Note the layering: the registry delegates its hard agreement problem to a File 08 consensus system.)

3.3Client-Side vs Server-Side Discovery#

3.4How Kubernetes Does It (the modern default)#

Kubernetes bundles discovery in: each Service gets a stable virtual IP + DNS name (payments.default.svc.cluster.local); etcd (File 08 consensus) is the registry storing which Pods back each Service (a Pod is Kubernetes' smallest deployable unit — one or more containers that share a network identity and live/die together; each running instance of your service is one Pod, and Pods are exactly the ephemeral, IP-changing things §3.1 warned about); kube-proxy (a per-machine agent that programs the node's network rules so traffic to a Service's virtual IP gets distributed to the backing Pods) or the mesh load-balances across the healthy Pods; readiness probes (File 01) gate which Pods receive traffic. So a service just calls http://payments by name and Kubernetes handles discovery + LB + health transparently.

4Resilience Patterns: Circuit Breakers, Retries, Bulkheads, Timeouts#

These patterns exist to contain partial failure and prevent cascading failure. This is the heart of microservice reliability.

4.1Timeouts (the non-negotiable foundation)#

Never wait forever. Every network call must have a timeout. Without one, a hung dependency holds your thread/connection indefinitely; enough hung calls and you exhaust the pool → you go down because they were slow. Set aggressive, sensible timeouts — based on the dependency's p99 latency (the 99th percentile: the value that 99% of requests are faster than; percentiles like p50/p95/p99 are how latency is always discussed, because the average hides the slow tail — a 10ms average with a 2s p99 means 1 in 100 users waits 2 seconds). A timeout just above p99 cuts off only the pathological 1%. Timeouts are the prerequisite for every other resilience pattern.

4.2Retries (with backoff + jitter + budget)#

Transient failures (a blip, a brief overload) often succeed on retry. But naive retries are dangerous: if D is overloaded, everyone retrying multiplies the load → retry storm → D never recovers (File 03). Rules:

4.3The Circuit Breaker (the signature pattern)#

Sub-Concept: Circuit Breaker. Named after an electrical circuit breaker that "trips" to prevent a fire. It wraps calls to a dependency and monitors failures; when failures exceed a threshold, it "opens" (trips) and immediately fails all further calls without even attempting them for a cooldown period — giving the struggling dependency room to recover and freeing the caller from wasting threads on doomed calls. It has three states:

- CLOSED (normal): calls pass through; the breaker counts failures. If the failure rate crosses a threshold (e.g., >50% of the last N calls fail), it trips to OPEN. - OPEN (tripped): calls fail fast immediately (return an error or a fallback — Section 4.5) without calling the dependency. After a cooldown timer (e.g., 30s), it moves to HALF-OPEN. - HALF-OPEN (testing): allows a few trial calls through. If they succeed, the dependency has recovered → back to CLOSED. If they fail, → back to OPEN (extend the cooldown).

Why it's essential: it converts a slow, cascading failure into a fast, contained one. Instead of 10,000 threads blocking on a dead service (which kills the caller too — cascading failure), the breaker trips and those calls fail instantly, keeping the caller alive and healthy. It also stops hammering the struggling service, letting it recover. Popularized by Netflix Hystrix; now in Resilience4j, Envoy, Istio.

4.4The Bulkhead Pattern#

Sub-Concept: Bulkhead. Named after a ship's watertight compartments — a hull breach floods only one compartment, not the whole ship. In software, you isolate resources (thread pools / connection pools) per dependency so that if one dependency goes bad and saturates its pool, it can't consume the resources needed to call other, healthy dependencies. Example: give recommendations its own 10-thread pool; if recommendations hangs, it exhausts only those 10 threads, and calls to payments (a separate pool) are unaffected. Without bulkheads, one slow dependency exhausts the shared thread pool and takes down calls to every dependency (cascading failure). Bulkheads contain the blast radius of a partial failure.

4.5Fallbacks & Graceful Degradation#

When a call fails (breaker open, timeout), return a fallback instead of an error when possible: a cached value (File 02), a default, a reduced experience. Example: if recommendations is down, show a generic "popular items" list instead of failing the whole page. Graceful degradation = the system loses features, not its life. (Amazon's page renders even if the recommendations widget is down.)

4.6Load Shedding & Rate Limiting (File 03, internal)#

Services protect themselves by shedding load when overwhelmed (reject low-priority requests, return 429) rather than accepting everything and collapsing. Combined with backpressure (File 07) and circuit breakers, this keeps a service alive and serving reduced traffic instead of dead.

4.7How They Combine (the resilience stack)#

Every service-to-service call is wrapped in:
  Timeout (never wait forever)
   └─ Retry (backoff + jitter + budget, idempotent + retryable only)
       └─ Circuit Breaker (trip on sustained failure -> fail fast)
           └─ Bulkhead (isolated pool -> contain the blast radius)
               └─ Fallback (degrade gracefully on failure)

Together they ensure a partial failure stays partial and never cascades into total collapse.

5Distributed Transactions: The Saga Pattern#

5.1The Problem: No ACID Across Services#

Database-per-service (2.2) means a business operation spanning services — "place order" = reserve inventory (inventory service) + charge payment (payment service) + create shipment (shipping service) — can't be one ACID transaction (they're separate databases). And 2PC (File 04) is a poor fit: it's blocking, hurts availability, doesn't scale, and many modern datastores don't support it. So how do you keep data consistent across services when you can't use a global transaction?

5.2The Saga Pattern#

Sub-Concept: Saga. A saga models a distributed transaction as a sequence of local transactions, one per service. Each local transaction updates its own database and triggers the next step (via a message/event, File 07). If any step fails, the saga runs compensating transactions — explicit "undo" operations — for all the previously completed steps, in reverse, to restore consistency. There's no global lock and no rollback of committed local transactions; instead you semantically undo them.

Sagas provide eventual consistency, not ACID isolation — for a window, the system is partially updated (e.g., inventory reserved but payment not yet charged). You accept that in exchange for availability and no distributed locks. Sagas require idempotency (File 07 — steps and compensations may be retried) and careful design of compensations (some actions are hard to undo — you can refund a payment, but you can't un-send an email; use semantic compensation like "send a correction").

Example — Order Saga (happy path then failure):

Step 1: Order Service     -> create order (PENDING)        | compensation: cancel order
Step 2: Inventory Service -> reserve items                 | compensation: release items
Step 3: Payment Service   -> charge card                   | compensation: refund
Step 4: Shipping Service  -> schedule shipment             | compensation: cancel shipment
        -> Order Service   -> mark order CONFIRMED

If Step 3 (payment) FAILS:
   run compensations in reverse: release items (undo 2), cancel order (undo 1).
   Result: system consistent again (order cancelled, inventory freed), eventually.

5.3Two Saga Styles: Choreography vs Orchestration#

Rule of thumb: short/simple sagas → choreography; complex/long-running workflows → orchestration (the explicitness pays off).

5.4Supporting Patterns#

6The Service Mesh (sidecars, data/control plane)#

6.1The Problem the Mesh Solves#

All those resilience patterns (Section 4), discovery (Section 3), mTLS, retries, and metrics were historically implemented as libraries inside each service (Netflix Hystrix/Ribbon/Eureka in Java). Problems: every language needs its own library (polyglot pain — File 10's whole point!), upgrading the resilience logic means redeploying every service, and business code is tangled with networking boilerplate. The service mesh moves all of this out of the application and into infrastructure.

6.2The Sidecar Pattern#

Sub-Concept: Sidecar Proxy. A proxy deployed alongside each service instance (in Kubernetes, a second container in the same Pod) that intercepts all network traffic into and out of the service. The service thinks it's making a normal localhost call; the sidecar transparently handles discovery, load balancing (File 01), retries/timeouts/circuit breaking (Section 4), mutual TLS encryption, and metrics/tracing collection — without the service code knowing. The dominant sidecar is Envoy (also the engine behind many API gateways and Istio). Because it's a separate process, it's language-agnostic — the same networking features work for a Python, Go, or Java service identically.

6.3Data Plane vs Control Plane (the mesh's two halves)#

6.4What the Mesh Gives You (for free, uniformly, across languages)#

6.5The Cost of a Mesh#

Not free: operational complexity (running Istio is famously involved), latency (every call now passes through two extra proxy hops — client sidecar + server sidecar), resource overhead (a sidecar per instance = more CPU/RAM), and a learning curve. For a handful of services, a mesh is over-engineering — libraries or a simpler setup suffice. Meshes pay off at dozens–hundreds of services across many teams/languages. (Newer "sidecar-less"/ambient meshes aim to cut the per-pod overhead.)

7Data Management & Observability#

7.1Inter-Service Communication Styles#

7.2Observability (you cannot operate microservices without it)#

Sub-Concept: The Three Pillars + Distributed Tracing. In a monolith, a stack trace tells you what happened. In microservices, one user request touches 20 services — you need observability to see across them: - Metrics: aggregated numbers (RPS, latency percentiles, error rates, saturation) — the Four Golden Signals (latency, traffic, errors, saturation) per service. (Prometheus/Grafana.) - Logs: structured, centralized (ELK/Loki), with a correlation ID so you can gather all logs for one request across all services. - Distributed Tracing: a trace ID propagated through every service call, so you can see the entire path of a single request across all services and where the time/errors went (Jaeger, Zipkin, OpenTelemetry). This is the essential tool for debugging latency and failures in a microservice system — it turns "the request was slow" into "service D's DB call took 800ms."

Without observability, a microservice architecture is an undebuggable black box. The mesh (6.4) provides much of this automatically.

8Pros, Cons & Trade-offs#

8.1Benefits#

8.2Costs & Trade-offs (be brutally honest — interviewers reward this)#

The senior verdict: Start with a well-structured monolith ("monolith-first"). Extract microservices only when organizational scale (many teams), differential scaling needs, or fault-isolation requirements justify the enormous added complexity. Microservices are a solution to an organizational scaling problem, not a default. Premature microservices kill startups.

9Interview Diagnostic Framework (Symptom → Solution)#

10Real-World Engineering Scenarios#

10.1Netflix — The Microservices Pioneer (and the whole bible in one system)#

Netflix runs hundreds of microservices and invented much of this playbook — and notice how it uses every prior file:

10.2Uber — From Monolith to ~Thousands of Services (and DOMA)#

10.3Amazon — "Two-Pizza Teams" & the Service Mandate#

11Interview Gotchas & Failure Modes#

11.1Classic Questions#

11.2Failure Modes & Mitigations#

12The Grand Synthesis: How All 10 Concepts Assemble#

This capstone exists to show that a real large-scale system is not ten separate topics — it's one machine built from all of them. Trace a single "user watches a video" request through the whole bible:

 1. USER hits the system
      -> DNS / Anycast routes to nearest CDN PoP ............................ [05 CDN, 06 consistent hash]
      -> Video segments served from the edge cache ......................... [05 CDN, 02 caching]
      -> Cache MISS pulls from S3 object storage origin .................... [09 object storage]

 2. The app/API request (play button, recommendations)
      -> Edge L4 LB -> API Gateway (L7): auth, TLS, RATE LIMIT ............. [01 LB, 03 rate limiting]
      -> Gateway routes east-west into the microservices .................. [10 microservices]

 3. Inside the mesh (service-to-service)
      -> Service DISCOVERY finds healthy instances ........................ [10 discovery, 08 consensus registry]
      -> Sidecar LOAD-BALANCES across them ................................ [01 LB, 06 P2C/consistent hash]
      -> Calls wrapped in TIMEOUT+RETRY+CIRCUIT BREAKER+BULKHEAD .......... [10 resilience, 03 backoff]

 4. Data
      -> Read-through CACHE (Redis) in front of the DB ................... [02 caching]
      -> DB is SHARDED across nodes (by user/key) ....................... [04 sharding, 06 consistent hash]
      -> Each shard REPLICATED via consensus (quorum, Raft) ............. [08 replication/consensus, CAP]
      -> "Recommendations down" -> FALLBACK to popular items ............ [10 graceful degradation]

 5. Async side-effects (record the view, update history, bill, ML)
      -> Emit events to KAFKA (transactional outbox) ................... [07 messaging, 10 outbox]
      -> Many consumer groups process independently, idempotently ..... [07 delivery guarantees]
      -> A cross-service "purchase" flows as a SAGA + compensations .... [10 saga, 04 no-2PC]

 6. Cross-cutting (always on)
      -> Distributed TRACING + metrics + logs across every hop ........ [10 observability]
      -> Everything replicated across AZs; failures self-heal ........ [08, 09, 01 health checks]

Every arrow is a concept from Files 01–09, orchestrated by the microservice/mesh patterns of File 10. That is system design: knowing not just each pattern, but how they compose into a resilient, scalable, observable whole — and which trade-offs you accept at each layer.

13Whiteboard Cheat Sheet#

 WHY: monolith walls are ORGANIZATIONAL (Conway's Law) - many teams can't share one deploy/DB/scale/runtime.
      Microservices = independent deploy + scale + fault-isolation + polyglot. COST: inherit all of Files 01-09.

 STRUCTURE:  Client -> API GATEWAY (north-south: auth, rate-limit[03], TLS, route, aggregate)
                        -> services (east-west) via SERVICE MESH
             Each service: own DB (database-per-service -> no cross-service ACID -> SAGA)

 DISCOVERY:  registry (Consul/etcd/Eureka, consensus-backed[08]) + health checks[01]
             client-side (Eureka+Ribbon) vs server-side (k8s Service + etcd + DNS)

 RESILIENCE (stop cascading failure from partial failure):
   TIMEOUT (never wait forever)
   RETRY (backoff+jitter+budget, idempotent+retryable only)   [03]
   CIRCUIT BREAKER: CLOSED --fail%-> OPEN(fail fast, cooldown) --> HALF-OPEN(trial) --> CLOSED/OPEN
   BULKHEAD (isolated pools -> contain blast radius)
   FALLBACK (graceful degradation: lose features, not life)

 DISTRIBUTED TXN: SAGA = local txns + COMPENSATIONS (reverse undo), eventual consistency, idempotent
   choreography (events, decentralized, simple/short) vs orchestration (coordinator, explicit/complex)
   Transactional OUTBOX + CDC solves dual-write. Event sourcing + CQRS optional.

 SERVICE MESH: SIDECAR proxy (Envoy) intercepts traffic -> discovery/LB/retry/breaker/mTLS/tracing
   DATA PLANE (sidecars carry traffic) + CONTROL PLANE (Istio configs them). Polyglot, no app changes.
   Cost: 2 extra hops latency + ops complexity + per-pod overhead. Worth it at dozens+ of services.

 OBSERVABILITY (mandatory): metrics(4 golden signals) + centralized logs + DISTRIBUTED TRACING (trace IDs)

 VERDICT: monolith-first; go micro only when ORG scale / differential scaling / fault isolation demand it.

One-sentence summary for an interviewer:

"Microservices split an app into independently-deployable, single-responsibility services (each with its own database) to let many teams move fast (Conway's Law) and scale/fail independently — but you inherit every distributed-systems problem: you find services via a consensus-backed registry, prevent cascading failure with timeouts + retry-budgets + circuit breakers + bulkheads + fallbacks, replace cross-service ACID with Sagas (compensating transactions, eventual consistency, transactional outbox), and push discovery/LB/mTLS/resilience/observability out of the code into a service mesh (Envoy sidecars = data plane, Istio = control plane); an API gateway guards north-south traffic while the mesh manages east-west, and distributed tracing is mandatory — so a real system is the composition of all 10 concepts, and you adopt this complexity only when organizational scale truly demands it."

🎓 The System Design Bible — Complete#

You now hold all 10 files, ordered from foundational to advanced:

Each concept builds on the last, and File 10 assembles them into one architecture. Master the trade-offs (every file's Pros/Cons + Symptom→Solution map) and the failure modes, and you can reason about any FAANG-scale system design problem from first principles.

End of File 10. End of the System Design Bible.