Understand how write-heavy storage engines transform random disk mutations into sequential NVMe appends via WAL and SkipList MemTables, while taming read and write amplification with Bloom filters and leveled K-way compaction.
architectureAct I · Scope, Workload & Back-of-the-Envelope Math
Durability Guarantees & NVMe Storage Envelope
Step 1 of 5
verifiedFunctional & Durability Goals
Core Engine
bolt
Sequential Write Path: Convert random key mutations into fast sequential WAL appends plus in-memory ordered SkipList inserts.
security
Crash-Safe WAL Recovery: Verify CRC32 checksums on reboot and replay uncommitted LSN records to reconstruct the active MemTable with 0% loss.
delete_forever
Tombstone Deletes & Range Scans: Record deletes as immutable 0x02 TOMBSTONE markers and support ordered prefix/range iterators across sorted SSTable runs.
merge
Zero-Downtime Compaction: Merge overlapping L0/L1 files in background threads and switch active reader views atomically via a versioned MANIFEST.
speedNon-Functional Constraints
Hardware Limits
hard_drive
NVMe Sequential vs. Random IOPS: Batch sequential writes via group commit to saturate NVMe bandwidth while avoiding random 4KB in-place page rewrites.
trending_up
10x Write Amplification Ceiling: Bound background compaction rewrites so SSD wear-leveling and foreground throughput stay sustainable over years.
layers
Bounded L0 File Count: Trigger compaction at 4 L0 files and apply backpressure (write stall) at 8 L0 files so point reads never scan dozens of overlapping tables.
timer
Predictable P99 Tail Latency: Keep P99 writes < 1.5 ms and eliminate unnecessary disk seeks on absent keys using in-RAM Bloom filters (≤1.0% FPR).
Peak Write Throughput
speed
150K WPS
~150 MB/s sequential WAL append at 1 KB/record using group-commit batching.
Active MemTable Cap
memory
64 MB
Freezes & flushes to L0 SSTable ~every 0.4s under sustained 150 MB/s peak load.
Bloom Filter Budget
filter_alt
10 bits / key
~1.0% False Positive Rate (7 hashes) → only ~1.2 MB RAM per 1M keys.
data_objectAct II · On-Disk Binary Formats & Storage Contracts
WAL Record Framing, SSTable Blocks & Atomic MANIFEST
Step 2 of 5
edit_documentWAL Record Frame & MemTable Entry
layersGroup-Commit Batching: Concurrent writer goroutines append their frames to a shared buffer and ride a single fdatasync() NVMe barrier, multiplying write throughput by 8–10x.
view_moduleSSTable Block Layout & Atomic MANIFEST
hubAct III · Interactive LSM-Tree Engine & Chaos Switchboard