Design a high-throughput distributed commit log powered by partitioned append-only segments, In-Sync Replica (ISR) quorum replication, OS kernel zero-copy (sendfile), and incremental cooperative consumer rebalancing.
architectureAct I · Scope & Back-of-the-Envelope Scale Math
Streaming Guarantees & Cluster Capacity
Step 1 of 5
checklistFunctional Requirements
Core Semantics
verified
At-Least-Once & Idempotent Delivery: Track (producer_id, epoch, seq) per partition so retried producer batches never create duplicate log entries.
format_list_numbered
Strict Per-Key Ordering: Route records via murmur2(order_id) % P so all events for the same entity land sequentially inside a single partition.
group_work
Horizontal Partition & Consumer Scaling: Scale throughput linearly by adding partitions and assigning disjoint subsets to consumer group workers with cooperative rebalance.
history
7-Day Tiered Retention & Replay: Retain immutable log segments for 7 days on NVMe/object storage so consumers can rewind offsets and replay streams deterministically.
shieldNon-Functional Constraints
Durability & Speed
security
Zero Committed Data Loss on Broker Crash: Enforce acks=all with min.ISR=2 across RF=3 so any committed offset (≤ HW) survives sudden leader termination.
bolt
Kernel Zero-Copy Fan-Out: Serve consumer fetches via sendfile() directly from OS Page Cache to NIC DMA buffers, bypassing broker heap allocation and GC pauses.
compress
End-to-End Batch Compression: Compress 64-record batches with Zstd on the producer and store/stream them compressed through brokers to cut disk & NIC usage by 4x.
report
Poison Pill Isolation (DLQ): Isolate malformed payloads into orders.dlq after bounded retries so a single bad message never permanently blocks partition progress.
Cluster Ingest
speed
1M msg/s
Sustained cluster ingest at 1 KB average payload across partitioned leaders.
warningUnsafe acks=1 Mode Active: Broker 1 ACKs immediately in 1.8 ms without waiting for ISR followers. If Broker 1 crashes before replication, acknowledged records are permanently lost!
schemaCluster Topology · Partitioned Append & ISR Quorum (acks=all)
PRODUCE BATCH
publish
PRODUCE BATCH
Loading cluster simulator...
verifiedInvariant active
electric_boltChaos Switchboard
0 Active
terminalLive Event Console
codeAct IV · Zero-Dependency Go Reference Blueprints