BACKEND.GLEEE.IN · MODULE 02 · VENDOR-AGNOSTIC 5-STEP PLAYBOOK

Consistent Hashing & Distributed Cache

Route 500K+ QPS across sharded in-memory clusters with minimal K/N key migration on node churn, stop thundering-herd DB stampedes using Single-Flight coalescing and XFetch, and neutralize viral hot keys.

Act I · Scope & Back-of-the-Envelope Scale Math

Sharded Caching Guarantees & Cluster Sizing

Step 1 of 5
Functional & Non-Functional Requirements
Cluster SLAs
  • O(log N) Deterministic Routing & Minimal Churn: Route keys via binary search over a sorted virtual-node ring so adding or losing a shard remaps only K / N keys instead of ~87% under modulo hashing.
  • < 5% Shard Skew Variance: Assign 150 vnodes/shard to smooth statistical ring gaps and prevent unbalanced RAM eviction across heterogeneous nodes.
  • Thundering-Herd Stampede Protection: Coalesce concurrent cache misses behind Single-Flight locks and refresh hot entries ahead of hard TTL via probabilistic early expiration (XFetch).
  • Viral Hot-Key Isolation & < 1.5 ms P99: Combine app-tier L1 W-TinyLFU caching with key salting (#0..#3) so a 120K QPS celebrity key never saturates a single shard NIC or CPU core.
Back-of-the-Envelope Scale Math
500K QPS Target
Peak Read QPS
500K QPS

Peak cluster read throughput (~62.5K QPS/shard across 8 shards with 2x headroom).

Working Set
256 GB RAM

8 Shards × 32 GB usable RAM (~250M entries @ ~1 KB avg value + hash overhead).

Hit Latency
< 1.5 ms

P99 cache hit: 0.1ms ring binary search + 0.9ms intra-AZ TCP + 0.4ms decode.

Churn Remap
K / N Keys

Only ~12.5%–25% keys migrate on node add/crash vs. ~87.5% under hash % N.

Act II · Wire Protocol & Ring / Cache Schema

gRPC / HTTP Cache Contract & Virtual Node Ring Layout

Step 2 of 5
Wire Protocol: gRPC / HTTP Cache Contract
Internal Ring, Salting & Coalescing Schema
Act III · Interactive Cluster Topology, Hash Ring & Chaos Switchboard

Consistent Hash Ring, Single-Flight & Hot-Key Shield

RING BALANCED
Step 1 / 6
Consistent Hash Ring · Hash Ring & Vnodes
XXHASH32
0 .. 2^32-1 450 Vnodes (3x150) O(log N) SEEK router App Router Single-Flight · L1 SF: READY L1 TinyLFU: OFF Shard A 33% · 150 vn CPU 32% · 84 GB Shard B 34% · 150 vn CPU 35% · 86 GB Shard C 33% · 150 vn CPU 33% · 86 GB Shard D STANDBY · 0 vn CPU 0% · Ready database Origin DB Primary SSOT 240 QPS · SAFE CPU 14% · OK
XXHASH32

Loading simulator...

Invariant active
Live Datacenter Event Console
Live Telemetry HUD
Real-Time
P99 Latency
1.1 ms
SLO < 1.5 ms
Cache Hit Ratio
99.6%
Target ≥ 99.2%
Shard Skew / Remap
2.8% variance
150 vnodes / shard
Origin DB QPS
240 QPS
Protected SSOT
Chaos Switchboard
0 Active
Act IV · Zero-Dependency Go Reference Blueprints

Idiomatic Go Contracts & Production Stack Mapping

Auto-Synced to Simulator
Production Stack Mapping Strip (Vendor-Agnostic → Real Systems) 6 Core Primitives
Vendor-Agnostic Primitive Production Stack Equivalents Operational Role
Consistent Hash Ring Envoy Maglev / Cassandra / Amazon DynamoDB O(log N) clockwise routing with K/N key migration on churn
Distributed KV Cache Redis Cluster / Dragonfly / Memcached Sub-millisecond sharded RAM working set (256 GB+)
Request Coalescing Go x/sync/singleflight / Varnish / NGINX proxy_cache_lock Collapses 400 concurrent misses into 1 leader DB query
Hot-Key Mitigation Redis hotkeys / Count-Min Sketch / Envoy Local Cache Detects viral skew and absorbs reads in L1 or salted replicas
Cache Eviction Policy W-TinyLFU (Caffeine) / Redis allkeys-lfu Frequency-aware admission resisting one-hit scan pollution
Ring Membership / Discovery etcd / Consul / SWIM Gossip Protocol Propagates live shard additions and failure detections cluster-wide
Act V · CAP / PACELC Tradeoffs & Production SLOs

Distributed Caching Tradeoffs & Reliability Targets

Step 5 of 5
Tradeoff 01 Topology Routing

Consistent Hash Ring vs. Fixed Hash Slots (16,384)

Hash Ring + Vnodes

Zero central slot map; any stateless router hashes keys locally in O(log V) and absorbs node churn cleanly with K/N remapping.

Fixed Hash Slots (16384)

Explicit slot ownership enables atomic multi-key hash-tags and controlled slot migration, but requires gossiped slot-map metadata.

Verdict Use Consistent Hash Ring + 150 Vnodes for stateless Memcached/Envoy pools; use Fixed Slots for stateful Redis Cluster migrations.
Tradeoff 02 Cache Pattern

Cache-Aside vs. Read-Through / Write-Through

Cache-Aside (Lazy)

Resilient to cache outages, only caches actually-read working sets, and lets app routers control Single-Flight and serialization.

Read / Write-Through

Simplifies application code and keeps cache warm on writes, but couples cache tier directly to DB schema and adds write latency.

Verdict Cache-Aside + Single-Flight is the default for high-scale microservices; pair with CDC invalidation when staleness windows must stay tight.
Tradeoff 03 Expiry Strategy

Hard TTL vs. Probabilistic Early Expiration (XFetch)

XFetch Early Refresh

As logical_expiry nears, 1 lucky reader triggers an async refresh while others still get sub-ms hits—eliminating synchronized stampedes.

Hard TTL Only

Simple to reason about, but when a popular key expires at second T, hundreds of concurrent callers miss simultaneously.

Verdict Combine XFetch (prevents cold-miss spikes on hot keys) with Single-Flight (hard backstop if a key is truly evicted).
Tradeoff 04 Hot-Key Defense

Key Salting (#0..#3) vs. Local L1 TinyLFU Cache

Local L1 TinyLFU (5s)

Absorbs 120K QPS in-process with zero network RTT and zero write amplification, trading up to 5s of eventual consistency.

Key Salting (#0..#3)

Spreads reads across multiple shards without local RAM staleness, but multiplies writes/invalidations by N salted replicas.

Verdict Use L1 TinyLFU (5s TTL) first for read-heavy viral keys; layer Key Salting when strict cross-node freshness is required.
Production Observability SLOs
Cluster Telemetry Targets
Cache Hit Ratio ≥ 99.2% L1 + Distributed Ring
Shard Skew Variance ≤ 5.0% 150 vnodes / shard
Stampede DB Fanout 1 req / key Single-Flight coalesced
P99 Hit Latency ≤ 1.5 ms Intra-AZ binary seek