Reliability — Outages & Congestion

Every part before this one added capability. Proof of History gave the network a clock, Sealevel gave it parallel execution, the validator pipeline wired those into a machine that ingests, orders, executes, and propagates transactions at hardware speed. This part asks the uncomfortable follow-up question the whole book has been circling: once you have built a single global state machine that runs that fast, what makes it fall over — and why is a speed-first chain more exposed to falling over than a slow one?

That is not a rhetorical question. Solana has halted — stopped producing blocks entirely — several times in its history, and has spent long stretches badly congested. Those events are not embarrassing footnotes to skip past; they are the most instructive thing in the book, because a system tells you what it truly optimized for by how it breaks. This part takes the outages seriously, traces each to a root cause in the architecture you have already learned, and then shows the specific fixes that bought robustness back.

The core tension: throughput trades away slack

Return to the book’s throughline: how do you build a single global state machine that runs at hardware speed without falling apart? The first half of that sentence — hardware speed — is exactly what makes the second half hard.

A slow chain has slack built in. Bitcoin produces a block roughly every ten minutes; Ethereum every twelve seconds. Between blocks, a full node is mostly idle, waiting. That idleness is wasted throughput, but it is also a shock absorber: a sudden flood of transactions has minutes or seconds of headroom to be absorbed, queued, and priced before the next block must be produced. The system is designed to run far below its own ceiling.

Solana made the opposite bet, the one traced back in scale up, not out: keep one synchronous chain and run it as close to the hardware’s limit as possible. Sub-second slots, thousands of transactions a second, fees that are fractions of a cent. That design leaves almost no slack. A validator running near its CPU, memory, and network ceiling has little headroom to absorb a surprise. When load spikes past what the machine can handle, there is nowhere for the excess to go — and a chain with no slack is far more likely to stall than one that was loafing along at 5% utilization.

   SLOW CHAIN (lots of slack)            SPEED-FIRST CHAIN (little slack)
   ──────────────────────────            ───────────────────────────────
   capacity ┤██                          capacity ┤████████████████
            │██                                   │████████████████
      load  ┤██          headroom                 │████████████████  ← running
            │██          absorbs                  │████████████████    near the
            └────────────  spikes                 └────────────────    ceiling
   a spike has minutes of room           a spike has milliseconds of room;
   before the next block                 past the ceiling, the machine stalls

This is the first-principles claim of the whole part, and it is worth stating bluntly: a system tuned for maximum throughput is, all else equal, more prone to halting under load than a slower one, because it kept less slack in reserve. That is not a bug in Solana. It is the price of the throughput bet — the reliability line item on the same invoice that bought the speed.

Two kinds of failure: liveness vs safety

Before we look at any incident, we need one distinction, because it is the difference between “annoying” and “catastrophic.” Distributed systems can fail in two fundamentally different ways.

A liveness failure means the system stops making progress. The chain halts: no new blocks, no confirmations, transactions pile up unprocessed. It is down. But nothing it already did is wrong — the ledger up to the halt is intact, and once operators restart from an agreed slot, it resumes. Liveness failures cost uptime.
A safety failure means the system produces an incorrect result. A confirmed transaction gets reversed, the ledger forks in a way that double-spends, or funds are minted from nothing. The chain kept running — but you can no longer trust what it says. Safety failures cost integrity, which for a money system is the thing you cannot afford to lose.

Hold this distinction, because it reframes Solana’s entire reliability history:

Solana’s failures have been overwhelmingly liveness failures — the network halted and was restarted — not safety failures. There is no widely-reported case of the protocol confirming-then-reversing transactions or minting funds from thin air during its outages.

That matters enormously for judging the trade-off. “The chain went down for a few hours and operators coordinated a restart” is a real, serious operational cost — but it is a far less severe failure mode than “the chain silently corrupted the ledger.” A well-designed system, if it must fail, should fail safe: stop rather than lie. Solana’s outages, uncomfortable as they were, are mostly the system doing exactly that. The Tower BFT and fork-choice machinery is what keeps the failures on the liveness side of the line.

The causal chain this part follows

The outages are not a grab-bag of unrelated bugs. They follow a pattern — a chain of cause and effect that runs straight through the architecture. This part walks that chain link by link, then walks back through the fixes.

   ROOT CAUSE                          →   the fix that addressed it
   ───────────────────────────────────    ─────────────────────────────

   no fee-market friction              →   local fee markets + priority
   (a failed tx costs ~nothing)            fees price contention per account

        ▼ so...

   spam amplification                  →   stake-weighted QoS gives real
   (bots retry endlessly, free)            validators a guaranteed share

        ▼ so...

   forwarding storms                   →   QUIC replaces raw UDP: connections,
   (duplicated retries flood the           backpressure, per-source limits
    expected leader)

        ▼ so...

   resource exhaustion                 →   the leader can shed load instead
   (CPU/memory/network saturate)           of drowning in it

        ▼ so...

   the network HALTS                       robustness bought back, incrementally

Read top to bottom, that is the anatomy of a Solana outage: because a failed transaction cost almost nothing, bots could spam without limit; because there was no global mempool, that spam was forwarded to the expected leader in duplicated storms; because the ingress path was raw UDP with no backpressure, those storms exhausted the leader’s resources; and an exhausted leader could not produce blocks, so the network stalled. Each fix in the right-hand column attacks one link. Together they are the story of how Solana traded a little raw throughput for a lot of robustness — and stopped halting.

The roadmap

Read the pages in order. Each takes one link in that chain, explains the mechanism from first principles, and grounds it in real, dated incidents. The middle pages diagnose; the later pages cure.

#	Page	The outage / cause / fix it teaches
2	Why a Speed-First Chain Can Halt	the first-principles reason no-slack throughput invites halting; liveness vs safety made concrete
3	No Fee Market, and How Spam Amplifies	the root cause: near-zero cost to fail meant no economic friction on spam, and how that amplifies
4	Forwarding Storms & Resource Exhaustion	how mempool-less forwarding turned spam into duplicated storms that exhausted the leader
5	The Outage Timeline — What Actually Happened	the real, dated incidents (Sept 2021 onward): trigger, mechanism, and restart, told honestly
6	The Ingress Fix — QUIC & Stake-Weighted QoS	replacing raw UDP with QUIC and prioritizing by stake to make ingress controllable
7	Pricing the Load — Local Fee Markets & Priority Fees	per-account fee markets and priority fees: putting an economic price on contention
900	Revision · Reliability	the whole causal chain and its fixes, compressed to what you must retain

By the end you will be able to take any Solana outage, place its trigger somewhere on that causal chain, name which link failed, and point at the fix that link received. You will also be able to say — precisely — why this class of failure is the bill for the throughput bet, not a flaw that a slower chain would have avoided for free.

The thread

What does studying the outages force you to understand? That reliability and throughput are not independent knobs — they are ends of the same trade-off. Every design choice that made Solana fast (no idle slack, no fee-market friction on failed transactions, no global mempool, a raw-UDP firehose at ingress) is also a choice about how the system behaves under overload. You cannot understand the halts without understanding the speed, and you cannot claim to understand the speed until you can explain the halts. This is the honest completion of the book’s throughline: running at hardware speed and not falling apart are the two halves of one engineering problem, and the fixes in this part are the years of work spent balancing them.

The first job is to make the core claim rigorous — why, mechanically, does a chain with no slack halt under load when a slow one merely gets slow? That is Why a Speed-First Chain Can Halt.

A note on the figures ahead

One discipline for this whole part, stated once here. The dates and durations you will read — “roughly seventeen hours,” “around a day” — are widely-reported approximations, not values pulled from an authoritative on-chain clock; treat the hour counts as ballpark. And the overall trend is unambiguous and important: network reliability improved markedly as the client matured. The outages cluster in the early years; the fixes in pages 6 and 7 are why. When this part discusses current behavior, assume “as of 2024–2025” and expect the numbers to keep moving as the client evolves. We describe mechanisms, which are durable, and hedge magnitudes, which are not.

Check your understanding

State, from first principles, why a chain tuned for maximum throughput is more prone to halting under load than a slower chain. What is the single word for what the slow chain has and the fast one gave up?
Distinguish a liveness failure from a safety failure. Which kind has Solana’s history overwhelmingly been, and why does that distinction matter for judging the throughput trade-off?
Lay out the causal chain this part follows, from the economic root cause to the network halting. Name at least four links in order.
Pick any one link in that causal chain and name the fix (from a later page) that was built to address it.
Why does this part insist on hedging the dates and durations of outages while stating the mechanisms confidently? What is the difference between the two?

Show answers

A slow chain runs far below its own capacity ceiling, so it carries slack — idle headroom between blocks that absorbs a sudden load spike by queuing and pricing it before the next block is due. A speed-first chain runs close to the hardware’s limit by design, so it has almost no headroom; when load spikes past the ceiling there is nowhere for the excess to go, and the machine stalls rather than merely slowing. The word is slack — the fast chain traded it away for throughput.
A liveness failure means the system stops making progress (the chain halts, but nothing it already did is wrong; a restart resumes it) — it costs uptime. A safety failure means the system produces an incorrect result (a reversal, a double-spend, minted funds) — it costs integrity, which a money system cannot afford. Solana’s history has been overwhelmingly liveness failures; it halted but did not corrupt the ledger. That matters because failing-safe (stop rather than lie) is a far less severe failure mode, so the outages cost uptime under load, not the trustworthiness of the money.
No fee-market friction (a failed transaction costs almost nothing) → spam amplification (bots retry endlessly for free) → forwarding storms (mempool-less forwarding duplicates that spam onto the expected leader) → resource exhaustion (the leader’s CPU, memory, and network saturate) → the network halts. (Any four in that order.)
Examples: no fee-market friction was addressed by local fee markets and priority fees (pricing contention per account); forwarding storms / resource exhaustion at ingress was addressed by QUIC and stake-weighted QoS (connections, backpressure, and a guaranteed share for staked validators). Either mapping is correct.
Because dates and durations are widely-reported approximations, not authoritative measurements — they are volatile magnitudes that drift and vary by source, so overstating them would be a false precision. The mechanisms (why no slack invites halting, how spam amplifies, why forwarding storms exhaust a leader) are durable causal facts that do not change with a retelling. The discipline is to state durable mechanisms confidently and hedge volatile numbers.