MODULE 04 · DISTRIBUTED LOG & STREAMING

Event Streaming Platform

Design a high-throughput distributed commit log powered by partitioned append-only segments, In-Sync Replica (ISR) quorum replication, OS kernel zero-copy (sendfile), and incremental cooperative consumer rebalancing.

Act I · Scope & Back-of-the-Envelope Scale Math

Streaming Guarantees & Cluster Capacity

Step 1 of 5
Functional Requirements
Core Semantics
  • At-Least-Once & Idempotent Delivery: Track (producer_id, epoch, seq) per partition so retried producer batches never create duplicate log entries.
  • Strict Per-Key Ordering: Route records via murmur2(order_id) % P so all events for the same entity land sequentially inside a single partition.
  • Horizontal Partition & Consumer Scaling: Scale throughput linearly by adding partitions and assigning disjoint subsets to consumer group workers with cooperative rebalance.
  • 7-Day Tiered Retention & Replay: Retain immutable log segments for 7 days on NVMe/object storage so consumers can rewind offsets and replay streams deterministically.
Non-Functional Constraints
Durability & Speed
  • Zero Committed Data Loss on Broker Crash: Enforce acks=all with min.ISR=2 across RF=3 so any committed offset (≤ HW) survives sudden leader termination.
  • Kernel Zero-Copy Fan-Out: Serve consumer fetches via sendfile() directly from OS Page Cache to NIC DMA buffers, bypassing broker heap allocation and GC pauses.
  • End-to-End Batch Compression: Compress 64-record batches with Zstd on the producer and store/stream them compressed through brokers to cut disk & NIC usage by 4x.
  • Poison Pill Isolation (DLQ): Isolate malformed payloads into orders.dlq after bounded retries so a single bad message never permanently blocks partition progress.
Cluster Ingest
1M msg/s

Sustained cluster ingest at 1 KB average payload across partitioned leaders.

Sequential Disk I/O
1.0 GB/s

Sequential append throughput leveraging OS Page Cache + Zstd batch compression.

Produce P99
< 5.0 ms

End-to-end produce latency with acks=all and min.ISR=2 quorum.

Replication Factor
RF = 3

1 Leader + 2 Followers across AZs — survives 1 broker loss with 0 committed loss.

Act II · Wire Protocol & On-Disk Segment Contract

Produce Batch Frame & Append-Only Segment Layout

Step 2 of 5
ProduceRequest / Batch Frame (Binary / JSON View)
On-Disk Partition Segment Layout
Act III · Interactive Cluster Topology, Telemetry HUD & Chaos Switchboard

ISR Quorum, Zero-Copy DMA & Cooperative Rebalance Lab

ISR QUORUM 3/3
Step 1 / 6
Ingest Throughput
980K msg/s
acks=all · 3.8 ms P99
ISR Quorum
[B1, B2, B3] · 3/3
min.ISR = 2 satisfied
HW / LEO
1041 / 1041
Partition P0 synced
Consumer Lag
0 msgs
SLO < 1,000 msgs
Cluster Topology · Partitioned Append & ISR Quorum (acks=all)
PRODUCE BATCH
publish Producer Batcher pid=42 · zstd · 64 rec acks=all · seq=1042 EPOCH 1 · READY Idempotent Producer call_split Key Partitioner murmur2(key) % 3 order_id -> P0 (B1) Strict Per-Key Order Metadata Cache v1 dns Broker 1 (P0 Leader) 00000000000001000.log LEADER · EPOCH 1 HW 1039 1040 1041 1042 1043 LEO OS PAGE CACHE → KERNEL DMA ZERO-COPY sendfile(): 0 user-space copies · .index sparse seek storage Broker 2 (AZ-b) Follower · ISR IN-SYNC · LEO 1041 Pull Fetch Loop Active storage Broker 3 (AZ-c) Follower · ISR IN-SYNC · LEO 1041 Pull Fetch Loop Active group_work Consumer Group order-workers C1 · Worker Assigned: [P0, P1, P2] C2 · Worker Standby / Joining... __consumer_offsets: 1042 report Dead Letter Queue Topic: orders.dlq 0 ISOLATED · CLEAN Headers: part, offset, err
PRODUCE BATCH

Loading cluster simulator...

Invariant active
Chaos Switchboard
0 Active
Live Event Console
Act IV · Zero-Dependency Go Reference Blueprints

Idiomatic Go Contracts & Production Stack Mapping

Auto-Synced to Simulator
Production Stack Mapping Strip 6 Core Primitives
Vendor-Agnostic Primitive Production Streaming Systems Operational Role
Partitioned Commit Log Apache Kafka / Redpanda / Apache Pulsar / Kinesis Immutable sequential segments with O(1) append & per-key ordering
Metadata & Leader Consensus KRaft (Kafka Raft) / Redpanda Raft / BookKeeper + ZK Elects partition leaders, tracks epochs, and enforces ISR membership
ISR Quorum Replication Kafka acks=all + min.insync.replicas / NATS JetStream R=3 Advances High Watermark strictly after quorum follower persistence
Kernel Zero-Copy Transfer Linux sendfile(2) / splice / Redpanda Seastar DMA Streams page-cache bytes straight to NIC without user-space copies
Cooperative Consumer Group Kafka CooperativeStickyAssignor / KIP-848 / Pulsar Key_Shared Reassigns partitions incrementally without stop-the-world pauses
Dead Letter Queue (DLQ) orders.dlq Topic / AWS SQS DLQ / Pulsar DeadLetterPolicy Quarantines poison pills with offset/stack-trace headers to unblock lag
Act V · Architectural Tradeoffs & Production Observability SLOs

Streaming Architecture Tradeoffs & Reliability Targets

Step 5 of 5
Tradeoff 01 Backpressure

Pull vs. Push Consumers (Backpressure & Batching)

Pull-Based Fetch

Consumers pull at their own pace with max_bytes batching—naturally preventing slow workers from OOM-crashing under bursts.

Push-Based Broker

Slightly lower idle-to-first-byte latency, but brokers must track per-consumer flow-control windows or overwhelm slow subscribers.

Verdict Pull with long-polling (fetch.min.bytes + fetch.max.wait.ms) delivers both low latency and effortless backpressure.
Tradeoff 02 Durability vs. Latency

acks=all + min.ISR=2 vs. acks=1

acks=all + min.ISR=2

Guarantees zero committed data loss across single-broker crashes (~3.8 ms P99) because followers persist before ACK.

acks=1 (Leader Only)

Cuts produce P99 to ~1.8 ms, but any crash on the leader before follower pull permanently drops acknowledged messages.

Verdict Always use acks=all + min.ISR=2 for orders, payments, and CDC; reserve acks=1 strictly for loss-tolerant clickstream metrics.
Tradeoff 03 Kernel I/O

Zero-Copy sendfile() vs. End-to-End TLS Overhead

sendfile() + kTLS

Direct Page Cache → NIC DMA skips user-space buffers and GC. Linux kTLS preserves zero-copy even with TLS encryption enabled.

User-Space TLS

Standard user-space TLS forces copying bytes into broker heap for encryption, raising CPU utilization by 25–40% at 1 GB/s.

Verdict Terminate TLS via kTLS or sidecar/NIC offload so brokers keep zero-copy DMA throughput at line rate.
Tradeoff 04 Scaling Limits

Partition Count Scaling vs. Ordering & Rebalance Cost

Right-Sized Partitions

Pick P = max(target_tput / single_part_tput, max_consumers) to balance parallelism, open file descriptors, and leader failover speed.

Over-Partitioning (10K+)

Fragments sequential disk writes into random I/O, inflates producer batch memory, and slows controller leader elections on broker crash.

Verdict Never increase partition count casually—changing P changes murmur2(key) % P and breaks historical per-key ordering.
Production Observability SLOs
Streaming Cluster Targets
Produce P99 < 5 ms acks=all · min.ISR=2
Under-Replicated Partitions 0 All RF=3 followers in-sync
Consumer Lag < 1,000 msgs DLQ isolates poison pills
Committed Data Loss 0.00% Strict HW quorum fence