How Jepsen tests distributed systems with fault injection
Learn how Jepsen generates concurrent histories, injects network faults, checks linearizability, and reports distributed system failures.
Lesson summary
Jepsen turns a distributed system promise into a fault filled concurrent history and checks whether any legal explanation survives.
- A Jepsen test connects deployment, clients, generated work, faults, history, and a checker on one control node.
- The control node prepares real database nodes so the experiment starts from a known cluster state.
- A generator schedules operations and faults while logical processes perform one operation at a time.
- Overlapping operation intervals create concurrency even though each individual process stays sequential.
- Every request becomes an invocation event and every known response becomes a completion event.
- Ok means success, fail means definite failure, and info preserves an outcome that may be unknown.
- The nemesis is another scheduled process, so faults overlap ordinary client traffic instead of pausing it.
- A partition removes communication paths and creates components that may observe different replica states.
- Crashes stop a process while pauses freeze it, so recovery and pending operations can differ.
- Healing a fault begins recovery, and Jepsen keeps observing how long convergence and availability take.
- A workload creates operations while a model defines the legal state transitions those operations should obey.
- Linearizability places each operation at one instant inside its interval while preserving real-time order.
- The checker searches candidate sequential orders and rejects every branch that breaks time or model rules.
- Moving a partition across one replication event can change replica state, the later read, and the checker verdict.
- A stale read after a completed write leaves no legal linearization point and therefore proves a safety violation.
- A Jepsen result is evidence from an experiment, combining safety checks with latency, availability, and artifacts.
Which jobs must a Jepsen test connect into one repeatable experiment?
A Jepsen test connects deployment, clients, generated work, faults, history, and a checker on one control node.
Why does Jepsen prepare the cluster before generating work?
The control node prepares real database nodes so the experiment starts from a known cluster state.
How can Jepsen create concurrency while each logical process stays sequential?
A generator schedules operations and faults while logical processes perform one operation at a time.
What makes two client operations concurrent?
Overlapping operation intervals create concurrency even though each individual process stays sequential.
Which two events preserve the full lifetime of a request?
Every request becomes an invocation event and every known response becomes a completion event.
Why does a timeout sometimes become info instead of fail?
Ok means success, fail means definite failure, and info preserves an outcome that may be unknown.
Why does Jepsen schedule the nemesis beside ordinary clients?
The nemesis is another scheduled process, so faults overlap ordinary client traffic instead of pausing it.
What can a network partition change without stopping every node?
A partition removes communication paths and creates components that may observe different replica states.
Why are a crash and a pause different recovery problems?
Crashes stop a process while pauses freeze it, so recovery and pending operations can differ.
Why does Jepsen keep observing after a fault is healed?
Healing a fault begins recovery, and Jepsen keeps observing how long convergence and availability take.
What separate jobs do the workload and model perform?
A workload creates operations while a model defines the legal state transitions those operations should obey.
Where may a linearizable operation take effect?
Linearizability places each operation at one instant inside its interval while preserving real-time order.
How does the checker eliminate impossible sequential explanations?
The checker searches candidate sequential orders and rejects every branch that breaks time or model rules.
Why can moving one partition boundary change the checker verdict?
The cut may block or allow write replication to n3. That changes the replica value, the later read, and whether any legal order survives.
Why does a stale read after a completed write prove a safety violation?
A stale read after a completed write leaves no legal linearization point and therefore proves a safety violation.
What does a complete Jepsen result preserve?
A Jepsen result is evidence from an experiment, combining safety checks with latency, availability, and artifacts.
Stage 1 of 18
Setup
Here is the entire mystery. Three replicas hold a shared register named x, and each copy begins at zero. We will disturb this small system, then judge only what its clients actually report.
Process p0 sends write one and receives ok. That reply is stronger than seeing a packet leave the client. The database has told p0 that the operation completed.
A network partition then isolates n3, which still shows zero. That alone proves nothing. Distributed systems are allowed to become slow or unavailable during a fault, and replicas may briefly hold different states.
Process p2 begins a new read only after p0 received ok, and isolated n3 answers zero. The result looks wrong because old state has appeared after the database acknowledged new state.
Jepsen keeps the four client events in their observed order. Before calling this a violation, we must learn what those events mean, which reorderings concurrency permits, and what promise the checker is enforcing.
Stage 2 of 18
Test Anatomy
Our suspicious write and read need more than a checker. Someone must create the cluster, choose operations, introduce the partition, and preserve exactly what happened without quietly changing the database itself.
A generator proposes work, then logical client processes use the database’s ordinary API. Jepsen stays outside the implementation, which matters because the evidence should resemble what a real application could know.
The nemesis cuts network links while those clients continue, and history collects their invocations and completions. Which component holds work, faults, recording, and analysis together as one repeatable experiment?
Which component coordinates the full Jepsen experiment?
The control node closes that loop. It prepares database nodes, schedules clients and faults, records the resulting history, then hands the evidence to a checker. None of those jobs requires it to become a replica.
This arrangement can reproduce a failure only if the starting cluster is known. If n3 was misconfigured before the test, its zero would describe a broken setup rather than the fault we meant to study.
Stage 3 of 18
Build the Cluster
So the control node rewinds n1, n2, and n3 to the declared baseline where x equals zero. The experiment should create its own starting state instead of inheriting whatever yesterday’s run left behind.
An operating-system adapter prepares each host. A database adapter installs software, writes configuration, starts processes, and joins the replicas. Teardown later removes that state so another run can repeat the same recipe.
A started process can still be absent from cluster membership or unable to replicate. Which signal is strong enough to let our clients begin a trustworthy history?
When is the cluster ready for trustworthy test traffic?
The gate opens only after every expected replica reports the required membership, leader, and replication readiness. A larger cluster therefore makes the gate stricter, not merely slower to draw.
Now x equals zero for a reason we can defend. The next problem is not choosing random commands. It is creating enough independent work for interesting timings to occur without letting one client contradict itself.
Stage 4 of 18
Generate Operations
The generator places our write and reads in a queue, but it does not send them all through one client. One sequential client would turn the test into a tidy script with almost no ordering ambiguity.
Instead, write one goes to p0, a safe read goes to p1, and the later read goes to p2. Each process performs one operation at a time, while different processes may advance independently.
Suppose p0 becomes stuck waiting for a reply. Waiting for every process would drain away the race we wanted, so how should the generator use the remaining lanes?
One logical process is blocked. What should the generator do next?
It assigns eligible work to a free process while p0 remains occupied. Three independent lanes keep the canonical p0, p1, and p2 operations alive without allowing two operations to overlap inside one process.
Adjust Concurrency from one lane to several. One lane forces a single story, while extra lanes create overlapping possibilities that the checker must eventually separate.
The cards now sit on named process lanes, yet we still cannot tell which operations were truly concurrent. For that, each card must expand from a scheduled item into the time it remained active.
Stage 5 of 18
Concurrent Clients
Stretching those same cards from invocation to completion gives us operation intervals. The identities have not changed, but time now shows information that a queue or printed log could not carry.
The p0 write and p1 read overlap. Neither finished before the other began, so a legal explanation may place either one first if the register responses still make sense.
A text log must print events one after another, even when requests were active together. Which evidence tells the checker that two operations genuinely had ordering freedom?
Which evidence proves that two operations were concurrent?
Their overlapping intervals permit more than one candidate order. Concurrency offers flexibility only where the intervals actually overlap. The p2 read has no such freedom relative to write one because its invocation appears after the write completion.
That distinction is the first constraint our eventual checker can trust. Jepsen preserves it by storing both ends of every interval, even when the request disappears between them.
Stage 6 of 18
Client to History
To understand how an interval enters history, follow p0’s write from the generator into a real database client. The planned operation becomes evidence only when the client begins interacting with the system.
Before sending the request, Jepsen records an invocation with the process, function, input, and start time. Even if every later packet vanishes, history still knows that this write began.
The server may apply the write just before the connection disappears. The client then lacks a reply, so which completion honestly preserves what might have happened?
The connection disappears before the reply. Which completion is honest?
A received success becomes ok. A vanished reply becomes info because the write may have happened without the client learning the result. That uncertainty enlarges the checker’s job, but shrinking it would falsify the evidence.
Switch Response between Success and Timeout. The request path stays the same, while its recorded ending changes from a known write to an operation that may still have taken effect.
We can now preserve an interval without pretending to know its outcome. The difference among ok, fail, and info matters enough to examine directly because each one permits a different later explanation.
Stage 7 of 18
History Outcomes
Every operation begins with invoke, while its ending states the strongest conclusion the client earned. These event types describe knowledge, not merely whether a screen should show green or red.
Ok says the operation succeeded. Fail says it definitely did not take effect, so a careful client adapter uses fail only when execution can be ruled out.
A timeout can arrive after another replica observed the write. Calling that fail would erase a possible state transition, so which ending keeps both explanations available?
The write times out after reaching the server. Which history outcome is honest?
Info keeps both possibilities alive. The checker may later complete that operation in whichever way makes the history legal, but it cannot simply delete the uncertainty for convenience.
Switch Outcome through Ok, Fail, and Info. The same invocation becomes a known transition, a ruled-out transition, or a branch the checker must continue carrying.
Now our history can admit what the client did not learn. The failures producing those gaps cannot happen in a quiet pause between workloads, because the interesting uncertainty appears when disruption catches operations in flight.
Stage 8 of 18
The Nemesis
Jepsen gives deliberate disruption its own logical process, called the nemesis. Its operations enter the generator’s schedule beside ordinary client work, so a fault has an observed start and finish.
A nemesis start can occur while reads and writes remain active. Because that overlap is recorded, a suspicious response can later be related to the exact fault window surrounding it.
Stopping every client before cutting the network would make the experiment orderly, but it would remove the races that create uncertain outcomes. How should the two schedules advance?
How should client traffic and nemesis faults be scheduled?
Client and nemesis schedules advance together, preserving the dangerous overlaps. Controlled means the test knows which fault it attempted and when, not that the database receives a gentle maintenance window.
Switch Fault among Partition, Crash, and Clock. The same pending request may arrive late, lose its server process, or become vulnerable to duplicate work under confused timing.
The first fault we need is a partition, because it challenges a tempting assumption. A machine that still answers requests may nevertheless be unable to learn what the rest of the cluster already knows.
Stage 9 of 18
Network Partitions
A network partition removes selected communication paths while n1, n2, and n3 may all remain alive. Liveness of a process therefore tells us nothing about which other replicas it can currently reach.
Before the cut, replication links let all three copies agree on x. After n3 becomes its own component, its local value can stop changing even while a client still reaches it.
That client connection proves n3 is responsive, not current. Which response protects a linearizable register when the isolated replica cannot confirm the latest completed write?
An isolated replica still receives a read. Which response keeps the history linearizable?
Rejecting the read sacrifices availability, while answering from stale state risks correctness. The partition shape also matters. A two-plus-one split retains a coordinating majority, while three isolated nodes remove every replica-to-replica path.
Switch Split between 2 + 1 and 1 + 1 + 1. Every process remains alive, yet the ability to coordinate disappears when no communication component contains a majority.
A partition changes reachability without ending a process. A client timeout can also come from the opposite situation, where the network remains intact but one process has crashed or stopped advancing.
Stage 10 of 18
Crash and Pause
Focus on n3 itself. A crash and a pause can both produce client timeouts, but they leave different machines behind, so identical observations may require different internal explanations.
A crash ends the process and its heartbeat. A pause freezes execution with memory still present. Resuming may continue old work, while restarting must rebuild from durable state and startup logic.
The timeout alone cannot distinguish them. Which combination of process state and retained memory tells us whether recovery should restart n3 or simply let it continue?
Which clue distinguishes a crash from a paused process?
The crash removes running state, while the pause leaves that state frozen in place. Recovery therefore follows a restart path for one fault and a resume path for the other.
Switch Failure between Crash and Pause. Compare the durable-state restart with a resumed process whose memory and pending request never disappeared.
Ending either fault only permits recovery to begin. The cluster may answer pings before its replicas agree again, so Jepsen must keep observing after the nemesis reports that healing is complete.
Stage 11 of 18
Heal and Observe
The nemesis restores communication while ordinary reads and writes continue. A quiet recovery period would hide the service users actually receive while replicas are catching up.
Nodes n1 and n2 already hold one, while n3 still holds zero. Repaired links can carry the missing state, but a healthy network does not make that transfer instantaneous.
The first successful ping proves only that transport returned. What later observation would show that the database, not merely the network, has actually recovered?
What proves that recovery restored correct service?
Recovery becomes meaningful when n3 receives one and later reads agree. Fast and slow catch-up produce different windows of stale or unavailable service even though both begin with the same healed link.
Switch Recovery between Quick and Slow. Communication is already restored, while the later reads reveal whether replica state is current or still catching up.
At last we have a fault-filled history, but events are not self-judging. Read zero might be legal for one kind of object and impossible for another, so the checker needs the promised behavior made explicit.
Stage 12 of 18
Workload and Model
Our history contains names such as write, read, and compare-and-set, but those names have no universal meaning. The checker first needs to know what object the clients were supposed to be using.
For a register, write one replaces the current value and read returns it. Compare-and-set changes the value only when its expected input matches, giving us rules for replaying a sequential story.
A sequence rejected by a register might be legal for a set or queue because those objects allow different transitions. Which declared rules bind this checker to the promise we care about?
What tells the checker which histories are legal?
The workload generates operations in the same vocabulary that the model interprets. Observed responses can now change model state or contradict it, so the verdict rests on declared laws instead of intuition.
Switch Workload between Register and Set. The operation sequence remains fixed, while changing the legal state transitions changes whether that same evidence can be accepted.
The model can judge one sequential order, but our overlapping clients never reported a single shared sequence. Linearizability must explain how a legal order can hide inside those intervals without rewriting real time.
Stage 13 of 18
Linearizability
Linearizability asks whether the concurrent history can appear to be one legal register sequence. Each operation receives a single imaginary effect point somewhere between its invocation and completion.
Points inside overlapping intervals may exchange order, which gives the checker legitimate flexibility. We do not need to know the database’s exact internal instant, only that some consistent choice exists.
Write one finishes before the final read even begins, so those intervals do not overlap. Can the checker still place that read before the completed write?
Can the final read be ordered before the completed write?
No. Real time forces the read after the write in every candidate order. Returning one can satisfy the register model there, while returning zero cannot be rescued by moving its point.
Switch History between Legal and Stale. The intervals never move, so the final response alone determines whether any legal set of effect points remains.
We can reason through this small history by eye, but Jepsen needs an algorithm that works when many operations overlap. It must explore every order still allowed by both time and the model.
Stage 14 of 18
Checker Search
The checker begins with register zero and a frontier of operations whose real-time predecessors are already placed. Choosing one frontier item extends a proposed sequential explanation by one step.
Overlapping operations create branches because either may come next. Real-time edges block impossible choices before replay begins, while the register model evaluates the state produced along each surviving branch.
Suppose a branch reaches a read returning a value that its own earlier writes never produced. Can any operation placed later repair that already contradictory prefix?
What should the checker do with a model-inconsistent branch?
No later choice can repair it, so that branch is pruned immediately. A valid history needs only one complete survivor. An invalid history is stronger because every allowable explanation has died.
Switch Overlap between More overlap and Less overlap. More overlap grows several candidate branches, while stronger real-time order can collapse the search to one forced path.
The checker now tells us what a history means, but the history itself came from one fault schedule. Moving the partition by a few moments can change replica state before the checker sees anything.
Stage 15 of 18
Move the Partition
Move the partition cut across the moment n3 applies write one. The same fault either blocks replication and preserves zero, or arrives after n3 is current, changing the later read and checker verdict.
Stage 16 of 18
Reveal the Violation
The timing experiment produced two histories that differ at one consequential place. In both, write one completed before p2 began reading. Only the value returned by that later read changes.
On the safe side, n3 returns one or refuses the read while isolated. Refusal may reduce availability, but neither outcome presents older state as current after an acknowledged write.
On the stale side, p2 begins after write one completed and still receives zero. Its effect point must remain inside that later interval, so where could a legal placement hide?
The stale read returns 0 after write 1 completed. What will the checker conclude?
Nowhere. Real time puts the read after the write, and the register model says the current value there is one. Every candidate order ends with the same contradiction.
A read that starts after write(1) completed cannot legally return the older value 0.
Try the Fault outcome control. Reject or fresh leaves a legal explanation, while Serve stale fixes zero inside the later interval and removes the checker’s last surviving branch.
This is why one small response can be decisive. The run did not merely look stressful. It produced a concrete history that the promised model cannot explain, which the final report must preserve without overclaiming.
Stage 17 of 18
Read the Result
Jepsen saves the invalid verdict beside the exact history, timeline, logs, latency measurements, and availability data. Another engineer can inspect the route from execution to conclusion instead of trusting a red badge.
The checker answers a safety question about the declared model. Throughput and latency describe how much service remained during faults and recovery. Neither result can replace the other.
An invalid history proves one implementation bug, but many valid histories cover only the executions that happened to run. How should a clean report describe that bounded success?
Every checked history is valid. What does that result establish?
A valid run means this experiment found no violation. It does not prove all possible executions correct. An invalid run is asymmetric because one preserved counterexample is enough to disprove a universal promise.
Switch Verdict between Valid run and Violation. The evidence bundle stays, while its honest claim changes from bounded absence of a finding to proof of one concrete bug.
The value of Jepsen is therefore not chaos or a mysterious pass badge. It is the chain connecting a controlled experiment, honest observations, an explicit model, and a reviewable conclusion.
Stage 18 of 18
Recap
We began with a suspicious stale read, then found the control node surrounding that incident with deployment, clients, faults, history, and a checker.
A repeatable verdict needed a known beginning, so Jepsen prepared every replica and waited until the cluster was ready.
The generator placed work on logical processes that stayed sequential alone but remained independent enough to create races together.
Those scheduled cards became intervals, revealing that overlap creates ordering freedom while a completed operation before a later invocation creates a hard real-time edge.
A client recorded each invocation and the most honest completion it could report, preserving the interval for later reasoning.
Ok, fail, and info then separated known success, known failure, and uncertainty, preventing a lost reply from being rewritten into a convenient answer.
The nemesis entered that same schedule, allowing a recorded partition, crash, pause, or clock fault to overlap the client operations it might disrupt.
A partition changed reachability, explaining how an isolated replica could answer from state it could no longer update.
Crashes and pauses produced similar timeouts but left different process state behind, so their recovery paths could not be treated as interchangeable.
Healing restored the possibility of communication, then continued observation revealed how long replica state and client-visible service took to recover.
The workload and model gave those events laws, so the checker could replay register operations instead of judging appearances.
Linearizability hid one atomic effect point inside each interval, preserving model rules while forbidding completed work from moving after later invocations.
The checker explored every ordering still permitted by those constraints and pruned a branch as soon as its own model state contradicted an observed response.
Moving the partition across replication changed n3 from zero to one, then changed the later read before search began.
With zero returned after write one completed, no legal point remained for the read, turning a small client response into a concrete safety counterexample.
The report preserved that proof beside its history, logs, latency, and availability, while a clean run would support only the narrower claim that this experiment found nothing.
The whole method now fits together. Jepsen makes the setup, fault, client knowledge, timing, model, and search precise enough that the system must explain the history it produced.