Metastable failure: why an outage outlasts a traffic spike
Trace identifiable requests, worker queues and connection reuse. Change LIFO to FIFO and explore how a metastable failure recovers under the same traffic.
Lesson summary
When an outage outlasts its trigger, find what keeps sending it more work.
- A reused connection keeps its server destination.
- Return order can bias the next batch toward a slow worker.
- Normal traffic can sustain an already degraded state.
- Changing the reuse rule can let existing capacity drain the queue.
R07 reuses C2. Can it now reach worker A?
An existing connection keeps its destination. Choosing C2 also chooses C.
A1 returns, then B1, then C1. Which does LIFO borrow next?
C1 returned most recently. LIFO uses return order, which need not favor the worker with spare capacity.
All burst requests have finished. Why does the queue keep growing?
Each completed pair puts C connections at the newest end again. Ordinary arrivals exceed the throughput of that repeated one-worker route.
What made the FIFO copy recover?
Arrivals and service times were fixed. FIFO let A and B serve queued requests instead of leaving that capacity idle.
Stage 1 of 6
Traffic is normal. Why are we still waiting?
The traffic spike is over. Requests are arriving at their usual rate again, yet the waiting line keeps growing. We might expect the service to recover as soon as the extra traffic disappears. Something inside the system is keeping the problem alive.
A worker processes requests. A connection is an open route to one worker; the client keeps idle connections in a pool so it can use them again. The queue holds work that has arrived but cannot run yet.
We will follow the requests, inspect what their connections leave behind, and change the rule that chooses the next route. This is an illustrative model of the feedback reported in OpenAI’s Habitat service. The request IDs and times are ours, not production measurements.
Stage 2 of 6
A request borrows an existing route
Start with one client and three workers: A, B and C. Each worker has two existing connections. R01 borrows A1, so it reaches A; R03 borrows C1 and reaches C. Reusing a connection later will keep that destination.
Our opening burst borrows all six connections at once. A worker runs one request at a time, even when several connections lead to it. The second request waits in its worker’s queue. C is temporarily paused, so its queue cannot advance yet.
By time four, A and B have finished their requests. Those four connections return to the client while C begins to catch up.
At time eight, C finishes too. Every request from the burst is now complete, but the six connections still exist.
From here on, ordinary traffic arrives as pairs every three time units. This client sends a pair together and waits for both responses before sending the next pair; later arrivals wait at the client. That batching assumption will matter, because each pair leaves an ordered set of connections for the next one.
Stage 3 of 6
What does the last response leave behind?
Keep a history of each response. The request finishes, but its connection survives and becomes available again.
A and B return four connections first. Appending each return builds the idle pool in completion order.
C returns at times six and eight, putting C1 and C2 at its newest end. LIFO means last in, first out: borrow the connection that returned most recently. It does not ask which worker has the most spare capacity.
With LIFO, the next pair borrows C2 and C1. R07 and R08 both reach C, so one runs while the other waits. A and B have reusable connections, but neither is selected.
Compare LIFO and FIFO with Reuse order. FIFO starts with the oldest idle connections; in this snapshot those reach A and B, so the pair can run together. That changes one dispatch. To understand the persistent outage, we need to follow what each completed pair leaves for the next one.
The slow worker returns connections last, and LIFO sends the next pair back to it.
Stage 4 of 6
Can normal traffic keep up?
Return to LIFO just after the burst has finished. C’s temporary pause is over, and all three workers now take two time units per request.
R07 finishes at eleven and R08 at thirteen. Two requests on one worker take four units, but another pair arrives every three. As the client waits for the second response, what happens to the line of arriving pairs?
If each pair takes four units but a pair arrives every three, what happens?
At time twenty-five, two requests are still waiting at the client. The last C connection returned just before the next dispatch, so LIFO chose C again. The completed work has recreated the selection that concentrated the work in the first place.
Continue to time forty-nine: six requests wait at the client while A and B have done no work since the opening burst. These are new requests. The original six are long gone.
The comparison without the opening trigger handles the same ordinary arrival rate without a growing client queue. The input can support healthy service, yet it can also sustain this degraded state once connection order has changed. That is the distinction behind a metastable failure: removing the trigger does not remove the feedback that now maintains the failure. Here the delay is fed back through the connection pool.
Stage 5 of 6
Change the next choice
To break that loop, take two copies of the state at time twenty-four. Both have completed thirteen requests, and both have four waiting at the client. Keep LIFO in one copy and change Reuse policy to FIFO in the other. The change applies to future borrowing; it does not cancel a request or erase either queue.
At time twenty-five, the current pair completes. LIFO chooses C again. Switching to FIFO borrows A1 and B1, so those workers start the next two requests in parallel. The previously idle capacity can finally help.
Continue with exactly the same arrivals and worker speeds. With FIFO, by time thirty-seven no requests are waiting at the client; keeping LIFO leaves four. C still receives work under FIFO. The aim is to stop every next pair being pinned there.
At time sixty, switching to FIFO gives forty completed requests, compared with thirty-one if we keep LIFO. Ten requests are waiting at the LIFO client’s gate; none are waiting at the FIFO gate, although two requests are still running. An empty waiting queue is recovery here, not an assertion that the system has stopped doing work.
Try both choices with Reuse policy. Each choice reruns the same intervention from time twenty-four, retaining the same past and arrivals. The difference comes from which idle connection is borrowed next. FIFO broke the reported Habitat loop; our batch timing and open connections make one version of that feedback visible, not a universal cure for every kind of overload.
Stage 6 of 6
Find what keeps the failure alive
A connection keeps its worker destination across requests. That is why the reuse decision can become a routing decision.
A slow response changes where its connection lands in the return order, so the next batch can be sent back to the same worker.
Repeated selection kept spare workers idle and allowed ordinary traffic to grow a queue after the burst had finished.
Changing that selection let the existing capacity drain the queue, without making any worker faster.
When an outage outlasts its trigger, look for the part of the system that keeps sending it more work. Other arrival patterns, pool limits, connection expiry, or a workload above total service capacity can change the result. The useful debugging question is what the slow response changes about future work.
Research behind the lesson
Three single-worker servers, six pinned connections, and a client that dispatches one batch at a time. Normal batches contain two requests and arrive every three time units. A six-request opening burst and a temporary pause on C seed the imbalance. All workers then need two units per request. No retries, idle expiry, new connections, or production timing are modeled.
- OpenAI: scaling Habitat (September 11, 2026)
The reported slow-process/LIFO feedback loop and FIFO mitigation. The account describes the pool behavior involved in that incident; it is not a claim about the default in every aiohttp version.
- Meta: a metastable failure state at scale (2014)
Correlated query bursts, return recency and persistent load imbalance. Its congested network links are a different bottleneck from the workers modeled here.