DriftlessDriftless

Distributed databases keep copies of data on several machines that must agree, even when networks split, machines crash and messages are lost. The bugs that break agreement are rare and depend on timing, so ordinary tests almost never see them, and when they do, they cannot reproduce them.

And a test suite that has never caught anything proves nothing.

I implemented Raft consensus in Go as a pure state machine, with no clocks, sockets or disks, so the same code runs in a real cluster and in a simulator.

A seeded simulator drives a five-node cluster through partitions, crashes, restarts and lost, duplicated or reordered messages while clients read and write, and every run replays exactly from its seed. A linearizability checker I wrote from scratch, with safety invariants, decides whether anything went wrong.

To prove the checks work, I planted three well-known Raft bugs that can be switched on.

Across 2,000 seeded runs of the correct build (27.6 million messages, 43,410 injected faults, 16,540 elections), no invariant broke and every history was linearizable.

Every planted bug was caught: stale reads in 291 of 2,000 runs, a forgotten vote in 20, and the Raft paper’s Figure 8 scenario in 40. CI fails if the simulator ever stops catching one.