The Engineer’s Field Guide to Load Balancers Load balancers look deceptively simple “split traffic across servers”, but they’re really the circulatory system of modern systems. They terminate connections, speak multiple protocols, apply routing policies, enforce security controls, measure health, and hide failure, all while keeping latency in the tens of milliseconds (or less) and throughput in the millions of requests per second. This guide goes deep: how load balancers actually work, the trade-offs behind common algorithms, what to tune for performance and reliability, and configuration patterns you can steal. What a Load Balancer Is (and Isn’t) At heart, a load balancer (LB) is a programmable network intermediary. As a reverse proxy, it accepts client traffic on a virtual IP (VIP), chooses a backend (“upstream”, “origin”, “target”), and forwards the traffic. Depending on where it operates in the stack: There are a few deployment archetypes: An LB is not automatically a WAF, API gateway, or service mesh; but modern L7 LBs overlap with those roles. The mental model that scales is control plane vs. data plane: the data plane is the hot loop that accepts, routes, and forwards packets; the control plane distributes configuration and membership (“these are the healthy instances, with these weights”). The Data Path: Step by Step When a request hits a reverse proxy, roughly this happens: Policies & Algorithms (and When They Bite) Choosing where to send traffic seems simple until tail latency shows up. Here’s what really matters. Round Robin / Weighted Round Robin. Evenly distributes requests; with weights, bigger machines do more. It ignores in-flight work and can overload a slow instance. Least Connections (or Least Requests). Prefers backends with fewer active requests. Better under heterogenous latency, but can be gamed by long-lived connections (e.g., WebSockets). Many implementations use EWMA of observed latency to approximate “least loaded”. Random with “Power of Two Choices”. Sample two backends at random and pick the less loaded. With negligible overhead, this dramatically reduces worst-case queueing. Consistent Hashing. Hash a key (user ID, session, cache key) to pick a backend; when membership changes, only a small fraction remaps. Variants: Rendezvous/HRW, Jump hash, Maglev. Great for caches and sticky state, risky if a shard gets hot. Sticky/Affinity. Keep a client on the same backend (via cookie, source IP, TLS session ID, or QUIC CID). Essential for stateful apps, but it undermines elasticity: a hot client can scorch a single node. Prefer stateless sessions or server-side stores when you can. Slow-start Warmup. After adding a backend, ramp traffic gradually while caches warm and JIT/GC settles. Helps avoid “cold start” spikes. Outlier Detection / Ejection. Temporarily remove backends that exceed error or latency thresholds. Combine with passive health checks (mark on 5xx/timeout) and active checks (HTTP/TCP probes). Connection Management: The Performance Bedrock Keep-alives & pools. Reusing connections to backends shaves RTTs and TLS handshakes. Size pools conservatively; too many idle connections starve ephemeral ports and memory. Idle timeouts. Clients, LBs, and servers each have their own. Mismatches cause mysterious disconnects (especially with gRPC or WebSockets). Align them and use heartbeats/pings. TCP specifics. HTTP/2 and HTTP/3. HTTP/2 multiplexes streams over one TCP connection (watch for server-side head-of-line at LB↔backend if only one connection is used). HTTP/3/QUIC runs over UDP; L4 LBs must use CID-aware routing for affinity because 5-tuple changes across NATs and migrations. WebSockets & long-lived streams. Treat separately: longer idle timers, periodic pings, and connection draining logic that waits for streams to finish. TLS at the Edge (or Not) Terminate at the LB to centralize ciphers, certificates, and HSTS/OCSP stapling. Then either: Practical details: Health Checking That Tells the Truth Active checks (HTTP/HTTPS/TCP) run on intervals with timeouts and “N-of-M” thresholds. Jitter intervals to avoid herding. Check the thing the user needs: for HTTP, hit /healthz that probes dependencies lightly; for gRPC use health-check protocol; for TCP ensure banner/readiness. Passive checks demote backends on real traffic errors/timeouts. Combine with outlier detection (500s, resets, slow EWMA) to protect users during partial failures. Drain & graceful shutdown. On deploy/scale-down, mark instances as draining so new requests go elsewhere while existing ones finish. Pair with slow-start when they come back. Reliability and High Availability No single LB. Use at least two per zone; for on-prem, float a VIP with VRRP (e.g., keepalived). In larger footprints, put LBs behind LBs (edge → L7 gateway → per-service mesh). Anycast + BGP. Advertising the same VIP from multiple sites pulls users to the closest/healthiest region. Requires robust health signalling to withdraw routes on trouble. State and stickiness. If you must do cookie/IP affinity at the LB, either (a) accept that failover loses stickiness, or (b) replicate stickiness state (more complexity), or (c) replace with consistent hashing so mapping is implicit. Retry storms. Clients retry, LBs retry, SDKs retry, suddenly a 1% error becomes a self-amplifying flood. Use bounded retries with exponential backoff + jitter, implement hedging carefully, and prefer timeouts that are shorter at the edge than in the client. Security at the LB Observability & SLOs Instrument the LB like a product: Define SLOs that separate LB time from backend time; you can’t fix what you can’t see. Kubernetes & Service Discovery In Kubernetes, a “load balancer” can be: Service discovery outside K8s often uses DNS (A/AAAA/SRV) with low TTLs, or a registry (Consul, Eureka, etcd/ZooKeeper). LBs either poll, watch, or receive pushed updates. Performance Engineering Hardware & NICs. Use RSS to spread interrupts; pin IRQs and worker threads to cores with NUMA awareness. Enable checksum offload, GRO/GSO/TSO, and large receive offload if compatible with your stack. Consider kTLS for in-kernel TLS data path, or DPDK/XDP for kernel-bypass when chasing 10–100 Gbit/s per node. Software loops. Event-driven reactors (epoll/kqueue/io_uring) with lock-free queues outperform thread-per-conn. Avoid thundering herd by using EPOLLET correctly or accept-mutex-like gating. Zero-copy. Prefer sendfile()/splice() and memory pools to cut copies and cache misses. Capacity math. Little’s Law: L = λ W. If your steady-state latency target W is 10 ms and you expect 100k RPS (λ), you’ll need ~1,000 concurrent in-flight requests (L) across the LB→backend leg per shard. Design headroom for P99 and failure scenarios (N-1 capacity). Plan for autoscaling lag, new instances need warmup. NAT port space. If the LB SNATs to backends, max simultaneous egress connections per destination is roughly the ephemeral port count (~30–60k per IP). Scale out source IPs or use full-proxy with connection pooling to avoid exhaustion. Routing Tricks You’ll Actually Use Failure Modes You’ll Meet in Real Life Practical Config Patterns Below are compact, representative snippets (sanitized to the essentials). They illustrate the ideas; adjust paths, certs, and limits to your environment. HAProxy: L7 HTTP with TLS termination, canary, health, and XFF. NGINX: WebSockets, rate limiting, and canary via header. L4 TCP with Proxy Protocol (preserve client IP through TLS pass-through). A Mini Checklist (pin to your runbook) Closing Thoughts A good load balancer isn’t just evenly splitting requests, it’s shaping traffic, absorbing failure, and giving you the levers to evolve your system without user pain. When in doubt, start simple (random/round-robin with health checks), instrument thoroughly, and move toward smarter policies only when the data shows you need them. And remember: the fastest way to make load balancing “easy” is to make your applications stateless and resilient, then the LB can do its best work.
