architecture
Act I · Scope & Back-of-the-Envelope Scale Math
Sharded Caching Guarantees & Cluster Sizing
Step 1 of 5
route
O(log N) Deterministic Routing & Minimal Churn: Route keys via binary search over a sorted virtual-node ring so adding or losing a shard remaps only K / N keys instead of ~87% under modulo hashing.
balance
< 5% Shard Skew Variance: Assign 150 vnodes/shard to smooth statistical ring gaps and prevent unbalanced RAM eviction across heterogeneous nodes.
call_merge
Thundering-Herd Stampede Protection: Coalesce concurrent cache misses behind Single-Flight locks and refresh hot entries ahead of hard TTL via probabilistic early expiration (XFetch ).
local_fire_department
Viral Hot-Key Isolation & < 1.5 ms P99: Combine app-tier L1 W-TinyLFU caching with key salting (#0..#3 ) so a 120K QPS celebrity key never saturates a single shard NIC or CPU core.
500K QPS
Peak cluster read throughput (~62.5K QPS/shard across 8 shards with 2x headroom).
256 GB RAM
8 Shards × 32 GB usable RAM (~250M entries @ ~1 KB avg value + hash overhead).
< 1.5 ms
P99 cache hit: 0.1ms ring binary search + 0.9ms intra-AZ TCP + 0.4ms decode.
K / N Keys
Only ~12.5%–25% keys migrate on node add/crash vs. ~87.5% under hash % N.
data_object
Act II · Wire Protocol & Ring / Cache Schema
gRPC / HTTP Cache Contract & Virtual Node Ring Layout
Step 2 of 5
hub
Act III · Interactive Cluster Topology, Hash Ring & Chaos Switchboard
Consistent Hash Ring, Single-Flight & Hot-Key Shield
RING BALANCED
0 .. 2^32-1
450 Vnodes (3x150)
O(log N) SEEK
router
App Router
Single-Flight · L1
SF: READY
L1 TinyLFU: OFF
Shard A
33% · 150 vn
CPU 32% · 84 GB
Shard B
34% · 150 vn
CPU 35% · 86 GB
Shard C
33% · 150 vn
CPU 33% · 86 GB
Shard D
STANDBY · 0 vn
CPU 0% · Ready
database
Origin DB
Primary SSOT
240 QPS · SAFE
CPU 14% · OK
donut_large
XXHASH32
Loading simulator...
verified
Invariant active
P99 Latency
timer
1.1 ms
SLO < 1.5 ms
Cache Hit Ratio
verified
99.6%
Target ≥ 99.2%
Shard Skew / Remap
balance
2.8% variance
150 vnodes / shard
Origin DB QPS
database
240 QPS
Protected SSOT
Shard C Crash (Node Churn)
OFF
Thundering Herd (400x Miss)
OFF
Viral Hot-Key (120K QPS)
OFF
add_circle
Trigger Live Node Add (Shard D Rebalance)
code
Act IV · Zero-Dependency Go Reference Blueprints
Idiomatic Go Contracts & Production Stack Mapping
Auto-Synced to Simulator
layers
Production Stack Mapping Strip (Vendor-Agnostic → Real Systems)
6 Core Primitives
Vendor-Agnostic Primitive
Production Stack Equivalents
Operational Role
Consistent Hash Ring
Envoy Maglev / Cassandra / Amazon DynamoDB
O(log N) clockwise routing with K/N key migration on churn
Distributed KV Cache
Redis Cluster / Dragonfly / Memcached
Sub-millisecond sharded RAM working set (256 GB+)
Request Coalescing
Go x/sync/singleflight / Varnish / NGINX proxy_cache_lock
Collapses 400 concurrent misses into 1 leader DB query
Hot-Key Mitigation
Redis hotkeys / Count-Min Sketch / Envoy Local Cache
Detects viral skew and absorbs reads in L1 or salted replicas
Cache Eviction Policy
W-TinyLFU (Caffeine) / Redis allkeys-lfu
Frequency-aware admission resisting one-hit scan pollution
Ring Membership / Discovery
etcd / Consul / SWIM Gossip Protocol
Propagates live shard additions and failure detections cluster-wide
balance
Act V · CAP / PACELC Tradeoffs & Production SLOs
Distributed Caching Tradeoffs & Reliability Targets
Step 5 of 5
donut_large
Tradeoff 01
Topology Routing
Consistent Hash Ring vs. Fixed Hash Slots (16,384)
check_circle Hash Ring + Vnodes
Zero central slot map; any stateless router hashes keys locally in O(log V) and absorbs node churn cleanly with K/N remapping.
tune Fixed Hash Slots (16384)
Explicit slot ownership enables atomic multi-key hash-tags and controlled slot migration, but requires gossiped slot-map metadata.
Verdict
Use Consistent Hash Ring + 150 Vnodes for stateless Memcached/Envoy pools; use Fixed Slots for stateful Redis Cluster migrations.
sync_alt
Tradeoff 02
Cache Pattern
Cache-Aside vs. Read-Through / Write-Through
check_circle Cache-Aside (Lazy)
Resilient to cache outages, only caches actually-read working sets, and lets app routers control Single-Flight and serialization.
tune Read / Write-Through
Simplifies application code and keeps cache warm on writes, but couples cache tier directly to DB schema and adds write latency.
Verdict
Cache-Aside + Single-Flight is the default for high-scale microservices; pair with CDC invalidation when staleness windows must stay tight.
timer
Tradeoff 03
Expiry Strategy
Hard TTL vs. Probabilistic Early Expiration (XFetch)
check_circle XFetch Early Refresh
As logical_expiry nears, 1 lucky reader triggers an async refresh while others still get sub-ms hits—eliminating synchronized stampedes.
tune Hard TTL Only
Simple to reason about, but when a popular key expires at second T, hundreds of concurrent callers miss simultaneously.
Verdict
Combine XFetch (prevents cold-miss spikes on hot keys) with Single-Flight (hard backstop if a key is truly evicted).
local_fire_department
Tradeoff 04
Hot-Key Defense
Key Salting (#0..#3) vs. Local L1 TinyLFU Cache
check_circle Local L1 TinyLFU (5s)
Absorbs 120K QPS in-process with zero network RTT and zero write amplification, trading up to 5s of eventual consistency.
tune Key Salting (#0..#3)
Spreads reads across multiple shards without local RAM staleness, but multiplies writes/invalidations by N salted replicas.
Verdict
Use L1 TinyLFU (5s TTL) first for read-heavy viral keys; layer Key Salting when strict cross-node freshness is required.
Cache Hit Ratio
≥ 99.2%
L1 + Distributed Ring
Shard Skew Variance
≤ 5.0%
150 vnodes / shard
Stampede DB Fanout
1 req / key
Single-Flight coalesced
P99 Hit Latency
≤ 1.5 ms
Intra-AZ binary seek