TRACK 04 · DISTRIBUTED SYSTEMS · MODULE 05

Consensus & Leader Election (Raft)

Understandable distributed consensus: randomized leader elections, quorum log replication, network partition survival, and monotonic fencing tokens.

Vendor-Agnostic 5-Step Playbook Go Reference Contracts 3 Interactive Scenarios 4 Chaos Drills
Act I · Scope, Requirements & Back-of-the-Envelope Scale Math

Consensus Guarantees & Quorum Sizing

Step 1 of 5
Functional Requirements
Linearizable Core
  • Linearizable Strongly-Consistent State Machine: Store cluster metadata, distributed locks, and service configuration with single-copy consistency.
  • Automatic Leader Election on Failure: Detect silent leader crashes via heartbeat timeouts and elect a new leader with the most up-to-date log automatically.
  • Ordered Log Replication Across N=5 Nodes: Replicate commands sequentially across a 5-node cluster and compact unbounded logs safely via snapshots.
  • Monotonic Fencing Tokens: Mint strictly increasing epoch tokens per leadership term so downstream external storage rejects delayed writes from zombie leaders.
Non-Functional Requirements
CP Safety & Failover
  • CP Under Network Partition (Zero Split-Brain): Minority partitions (2/5) must never commit writes; at most one leader can commit entries in any given term.
  • Survive f = 2 Node Failures: With N = 2f + 1 = 5 nodes deployed across availability zones, the cluster tolerates 2 simultaneous crashes or 1 maintenance + 1 outage.
  • Deterministic Failover < 300 ms: Randomized election timeouts (150–300 ms) break symmetry cleanly to avoid repeated split votes.
  • Crash Recovery via Persistent WAL + fsync: Persist currentTerm, votedFor, and log entries to NVMe before replying to any RPC.
Fault Tolerance
N = 5 (Quorum 3)

2f + 1 = 5 tolerates up to 2 crashed or partitioned nodes without losing availability.

Election Timeout
150–300 ms

3x–6x the 50 ms heartbeat interval; randomization prevents split-vote livelocks.

Quorum Commit P99
< 8.0 ms P99

1 RTT to the 2 fastest followers + NVMe group fsync (ignores slow 4th/5th tail).

Batched Throughput
25K ops/s

Pipelined AppendEntries with WAL group commit for metadata & lock workloads.

Act II · Wire Protocol (RPCs) & Persistent State Schema

Raft RPC Contracts, On-Disk WAL & Fencing Guard

Step 2 of 5
Raft Wire RPCs (raft.proto)
Persistent On-Disk State & Fencing Contract
Act III · Interactive 5-Node Raft Cluster & Chaos Switchboard

Leader Election, Quorum Replication & Fencing Token Lab

QUORUM 5/5 HEALTHY
Step 1 / 6
Active Term & Leader
Term 3 · N1 (Down)
Heartbeats halted
Quorum Health
4 / 5 Online (Need 3)
Majority achievable
Commit Index
Idx 11 (Applied)
State machine synced
Max Fencing Token
Token #3 · Safe
Monotonic epoch guard
5-Node Raft Cluster · Leader Election & Split-Vote Guard
LEADER CRASH
PARTITION (2 vs 3) Client Lock / KV App IDLE N1 T:3 DOWN 11:T3 — — N2 T:3 FOLLOWER 11:T3 — — N3 T:3 FOLLOWER 11:T3 — — N4 T:3 FOLLOWER 11:T3 — — N5 T:3 FOLLOWER 11:T3 — — External Storage Fenced Lock Resource Max Token: #3 READY · GUARDED
LEADER CRASH

Loading Raft cluster simulator...

Invariant active
Chaos Switchboard
0 Active
Live Datacenter Event Console
Act IV · Zero-Dependency Reference Blueprints (Go)

Idiomatic Go Consensus Core & Production Stack Mapping

Auto-Synced to Simulator
Production Stack Mapping Strip 6 Core Primitives
Agnostic Primitive Real-World Systems Key Engineering Note
Raft Consensus Engine etcd (Go) / HashiCorp Raft / TiKV (Rust) Provides strongly consistent metadata, leader election, and replicated logs
Distributed Lock & Lease etcd Leases / ZooKeeper Ephemeral ZNodes Auto-expires locks when holder crashes; must pair with fencing tokens
Persistent WAL & State Machine BoltDB / RocksDB / Pebble fsyncs currentTerm, votedFor, and log entries before RPC reply
Log Compaction InstallSnapshot RPC / Raft Snapshots Truncates applied WAL history and hydrates lagging or brand-new followers
Fencing Token Storage S3 Conditional Put / Postgres Version Check Rejects stale writes with 412 Precondition Failed when token < max_seen
Linearizable Reads Raft ReadIndex / Leader Leases ReadIndex verifies leadership via 1-RTT quorum heartbeat; immune to clock skew
Act V · CAP / PACELC Tradeoffs & Production SLOs

Consensus Tradeoffs & Production Observability

Step 5 of 5
Tradeoff 01 CAP: CP

CP over AP (Why Minority Partitions Must Stall)

CP Quorum Stall

Minority {N1, N2} cannot reach 3/5 votes and stalls writes cleanly, preserving strict linearizability on heal.

AP Minority Writes

Accepting writes on both sides of a partition creates conflicting lock owners and unresolvable state divergence.

Verdict For locks, leader election, and metadata, availability on the minority side must be sacrificed to guarantee zero split-brain commits.
Tradeoff 02 Linearizable Reads

ReadIndex vs. Leader Leases (Clock Skew Danger)

Raft ReadIndex (1-RTT)

Leader confirms authority with a lightweight quorum heartbeat before serving reads—100% immune to NTP jumps and VM pauses.

Leader Leases (0-RTT)

Serves local reads with zero RTT while lease timer holds, but clock drift or a 400ms GC pause can serve stale reads from a deposed leader.

Verdict Default to ReadIndex for strict correctness; only enable Leader Leases with bounded hardware clocks (e.g. TrueTime/PTP).
Tradeoff 03 Log Growth

Log Compaction & InstallSnapshot RPC

Periodic Snapshots

Compacts applied state into a point-in-time snapshot, truncating old WAL segments and catching up new/lagging nodes in seconds.

Unbounded Log Replay

Simple to implement, but causes gigabyte WAL bloat and forces newly added replicas to replay millions of historical entries.

Verdict Take asynchronous copy-on-write snapshots every ~10K–100K entries and stream them via InstallSnapshot RPC to lagging followers.
Tradeoff 04 End-to-End Safety

Why Distributed Locks Need Fencing Tokens

Monotonic Fencing Token

Downstream storage compares token >= max_seen atomically, rejecting delayed writes from GC-paused zombie clients with 412.

Lease-Only Lock

Even with a perfect Raft lock service, a client paused by GC right before writing to storage can wake up after lease expiry and corrupt data.

Verdict Consensus alone cannot stop a paused client—end-to-end safety requires downstream storage to validate monotonic fencing tokens.
Production Observability SLOs
Consensus Cluster Targets
Leader Election P99 < 300 ms Randomized 150–300 ms timer
Quorum Commit P99 < 10 ms 1 RTT + NVMe group fsync
Split-Brain Commits 0 Strict majority + fencing
Cluster Availability (N=5) 99.99% Survives f = 2 failures