TRACK 04 · DISTRIBUTED SYSTEMS · MODULE 05
Consensus & Leader Election (Raft)
Understandable distributed consensus: randomized leader elections, quorum log replication, network partition survival, and monotonic fencing tokens.
layers
Vendor-Agnostic
timeline
5-Step Playbook
code
Go Reference Contracts
hub
3 Interactive Scenarios
electric_bolt
4 Chaos Drills
architecture
Act I · Scope, Requirements & Back-of-the-Envelope Scale Math
Consensus Guarantees & Quorum Sizing
Step 1 of 5
verified
Linearizable Strongly-Consistent State Machine: Store cluster metadata, distributed locks, and service configuration with single-copy consistency.
how_to_vote
Automatic Leader Election on Failure: Detect silent leader crashes via heartbeat timeouts and elect a new leader with the most up-to-date log automatically.
sync_alt
Ordered Log Replication Across N=5 Nodes: Replicate commands sequentially across a 5-node cluster and compact unbounded logs safely via snapshots.
vpn_key
Monotonic Fencing Tokens: Mint strictly increasing epoch tokens per leadership term so downstream external storage rejects delayed writes from zombie leaders.
gpp_good
CP Under Network Partition (Zero Split-Brain): Minority partitions (2/5 ) must never commit writes; at most one leader can commit entries in any given term.
dns
Survive f = 2 Node Failures: With N = 2f + 1 = 5 nodes deployed across availability zones, the cluster tolerates 2 simultaneous crashes or 1 maintenance + 1 outage.
timer
Deterministic Failover < 300 ms: Randomized election timeouts (150–300 ms ) break symmetry cleanly to avoid repeated split votes.
hard_drive
Crash Recovery via Persistent WAL + fsync: Persist currentTerm , votedFor , and log entries to NVMe before replying to any RPC.
N = 5 (Quorum 3)
2f + 1 = 5 tolerates up to 2 crashed or partitioned nodes without losing availability.
Election Timeout
hourglass_top
150–300 ms
3x–6x the 50 ms heartbeat interval; randomization prevents split-vote livelocks.
< 8.0 ms P99
1 RTT to the 2 fastest followers + NVMe group fsync (ignores slow 4th/5th tail).
25K ops/s
Pipelined AppendEntries with WAL group commit for metadata & lock workloads.
data_object
Act II · Wire Protocol (RPCs) & Persistent State Schema
Raft RPC Contracts, On-Disk WAL & Fencing Guard
Step 2 of 5
hub
Act III · Interactive 5-Node Raft Cluster & Chaos Switchboard
Leader Election, Quorum Replication & Fencing Token Lab
QUORUM 5/5 HEALTHY
Active Term & Leader
crown
Term 3 · N1 (Down)
Heartbeats halted
Quorum Health
dns
4 / 5 Online (Need 3)
Majority achievable
Commit Index
format_list_numbered
Idx 11 (Applied)
State machine synced
Max Fencing Token
vpn_key
Token #3 · Safe
Monotonic epoch guard
PARTITION (2 vs 3)
Client
Lock / KV App
IDLE
N1
T:3
DOWN
11:T3
—
—
N2
T:3
FOLLOWER
11:T3
—
—
N3
T:3
FOLLOWER
11:T3
—
—
N4
T:3
FOLLOWER
11:T3
—
—
N5
T:3
FOLLOWER
11:T3
—
—
External Storage
Fenced Lock Resource
Max Token: #3
READY · GUARDED
how_to_vote
LEADER CRASH
Loading Raft cluster simulator...
verified
Invariant active
Network Partition (2 vs 3)
OFF
Leader GC Pause (+400ms)
OFF
Slow Follower Disk (N5 +250ms)
OFF
crisis_alert
Simulate Split-Brain & Partition Heal
code
Act IV · Zero-Dependency Reference Blueprints (Go)
Idiomatic Go Consensus Core & Production Stack Mapping
Auto-Synced to Simulator
layers
Production Stack Mapping Strip
6 Core Primitives
Agnostic Primitive
Real-World Systems
Key Engineering Note
Raft Consensus Engine
etcd (Go) / HashiCorp Raft / TiKV (Rust)
Provides strongly consistent metadata, leader election, and replicated logs
Distributed Lock & Lease
etcd Leases / ZooKeeper Ephemeral ZNodes
Auto-expires locks when holder crashes; must pair with fencing tokens
Persistent WAL & State Machine
BoltDB / RocksDB / Pebble
fsyncs currentTerm, votedFor, and log entries before RPC reply
Log Compaction
InstallSnapshot RPC / Raft Snapshots
Truncates applied WAL history and hydrates lagging or brand-new followers
Fencing Token Storage
S3 Conditional Put / Postgres Version Check
Rejects stale writes with 412 Precondition Failed when token < max_seen
Linearizable Reads
Raft ReadIndex / Leader Leases
ReadIndex verifies leadership via 1-RTT quorum heartbeat; immune to clock skew
balance
Act V · CAP / PACELC Tradeoffs & Production SLOs
Consensus Tradeoffs & Production Observability
Step 5 of 5
gpp_good
Tradeoff 01
CAP: CP
CP over AP (Why Minority Partitions Must Stall)
check_circle CP Quorum Stall
Minority {N1, N2} cannot reach 3/5 votes and stalls writes cleanly, preserving strict linearizability on heal.
tune AP Minority Writes
Accepting writes on both sides of a partition creates conflicting lock owners and unresolvable state divergence.
Verdict
For locks, leader election, and metadata, availability on the minority side must be sacrificed to guarantee zero split-brain commits.
schedule
Tradeoff 02
Linearizable Reads
ReadIndex vs. Leader Leases (Clock Skew Danger)
check_circle Raft ReadIndex (1-RTT)
Leader confirms authority with a lightweight quorum heartbeat before serving reads—100% immune to NTP jumps and VM pauses.
tune Leader Leases (0-RTT)
Serves local reads with zero RTT while lease timer holds, but clock drift or a 400ms GC pause can serve stale reads from a deposed leader.
Verdict
Default to ReadIndex for strict correctness; only enable Leader Leases with bounded hardware clocks (e.g. TrueTime/PTP).
inventory_2
Tradeoff 03
Log Growth
Log Compaction & InstallSnapshot RPC
check_circle Periodic Snapshots
Compacts applied state into a point-in-time snapshot, truncating old WAL segments and catching up new/lagging nodes in seconds.
tune Unbounded Log Replay
Simple to implement, but causes gigabyte WAL bloat and forces newly added replicas to replay millions of historical entries.
Verdict
Take asynchronous copy-on-write snapshots every ~10K–100K entries and stream them via InstallSnapshot RPC to lagging followers.
vpn_key
Tradeoff 04
End-to-End Safety
Why Distributed Locks Need Fencing Tokens
check_circle Monotonic Fencing Token
Downstream storage compares token >= max_seen atomically, rejecting delayed writes from GC-paused zombie clients with 412 .
tune Lease-Only Lock
Even with a perfect Raft lock service, a client paused by GC right before writing to storage can wake up after lease expiry and corrupt data.
Verdict
Consensus alone cannot stop a paused client—end-to-end safety requires downstream storage to validate monotonic fencing tokens .
Leader Election P99
< 300 ms
Randomized 150–300 ms timer
Quorum Commit P99
< 10 ms
1 RTT + NVMe group fsync
Split-Brain Commits
0
Strict majority + fencing
Cluster Availability (N=5)
99.99%
Survives f = 2 failures