Revision · Reliability

Every earlier part of this book optimized one thing: throughput. Proof of History removed the conversation about time; the account model made footprints knowable; Sealevel turned those footprints into parallel execution; the validator pipeline turned the validator into an assembly line that never waits on itself. Each of those was a bet on speed. This part was the bill for those bets coming due.

The throughline of the whole book is one question: how do you build a single global state machine that runs at hardware speed without falling apart? Every earlier part answered the first half — hardware speed. This part was the second half — without falling apart — and it was honest that, for years, Solana did fall apart, repeatedly, under load. This page walks the whole arc back: why a speed-first chain is structurally exposed, the exact causal chain that took the network down, the dated incidents where it happened, and the four fixes that bought robustness back. Hold it as one story, because it is one.

The structural exposure: speed leaves no slack

Start with why the failures were even possible. A system’s robustness under load is, roughly, its slack — the buffers, queues, and shock absorbers that let it absorb a surge without collapsing. Solana’s entire design philosophy is to remove slack, because slack is latency, and latency is the enemy of throughput. That trade is deliberate and it is the root of everything this part described.

Three design choices, each covered in Why a Speed-First Chain Can Halt, together removed almost all of the slack a chain normally has:

• No mempool. Most chains hold pending transactions in a global mempool — a buffer that also prices admission, because block space is auctioned out of it. Solana deletes the mempool (Gulf Stream forwards straight to the known leader). The buffer is gone, so a surge lands directly on the leader’s ingest stages with nothing in front of it.
• Synchronous, tightly-coupled execution. The pipeline runs at hardware speed precisely because its stages never wait. That means there is no back-pressure valve: if one stage saturates, the whole machine backs up behind it rather than gracefully shedding load.
• Near-zero, flat fees. Fees were a fixed handful of lamports per signature, unrelated to demand. Cheap transactions are a feature for users and a vulnerability for the network, because they make flooding almost free.

The crucial thing this part established is which kind of failure this exposure produces. Solana’s outages were liveness failures, not safety failures. The chain never forked, never let a double-spend through, never agreed on two conflicting histories — Tower BFT held the whole time. What broke was progress: the network stopped producing blocks. A safety failure corrupts the ledger; a liveness failure freezes it. Solana only ever suffered the second kind, and it suffered it because a chain with no slack has no way to stay live through a load it cannot keep up with.

    a chain's slack = its shock absorbers under load
    ───────────────────────────────────────────────
    mempool           buffers + prices a surge   ── Solana: deleted
    async queues      decouple fast/slow stages  ── Solana: minimized
    demand-based fees  make flooding expensive    ── Solana (early): flat, ~free

    remove all three  →  a surge hits the leader directly, for free,
                         with nothing to absorb or price it
                         →  liveness (not safety) failure under load

The causal chain: from free fees to a coordinated restart

The heart of this part was a single mechanism, built up one link at a time. It is worth reciting as a chain, because each link is a consequence of the one before it, and each of the fixes later attacks a specific link.

Near-zero flat fees → spam amplification. Covered in No Fee Market, and How Spam Amplifies, the first link is economic. When a transaction costs a fixed sliver of a cent no matter how contested the block is, the rational move during a hot event — an IDO, a popular NFT mint — is to send the same transaction hundreds or thousands of times to raise your odds of landing. Bots did exactly this. There was no rising price to say “stop”; a flat fee is the same whether the network is idle or on fire. Demand that a fee market would have priced instead got amplified, because the cheapest way to compete for scarce block space was raw volume.

Spam amplification → forwarding storms. Covered in Forwarding Storms & Resource Exhaustion, the second link is systemic. Gulf Stream’s forwarding discipline, which is a virtue at normal load, becomes a multiplier under a flood. Every node that receives a transaction forwards it to the upcoming leaders. Retries and RPC fan-out mean one user’s transaction becomes many packets on the wire, and the whole cluster amplifies the storm on the sender’s behalf. The old transport was raw UDP, which is connectionless: a validator receiving a firehose of UDP packets has no cheap way to tell whose they are or to throttle any one source. Every packet looks equally anonymous and equally urgent.

Forwarding storms → resource exhaustion. The flood of forwarded packets lands on the leader’s narrowest pipeline stages — fetch and sig-verify. Signature verification is real CPU work, and the leader must attempt it on packets before it can even discover most of them are junk or duplicates. The scarce resource — CPU, memory, the fixed time budget of a ~400 ms slot — gets consumed by spam, starving the legitimate transactions mixed into the same firehose.

Resource exhaustion → a coordinated-restart halt. When enough validators fall behind — unable to keep up with the incoming load, running out of memory, or failing to reach consensus on the tip within the slot’s time budget — the cluster stops making forward progress. Because the failure is loss of liveness, the recovery is not a software rollback of a corrupted ledger; it is the validator operators coordinating out-of-band to restart the cluster from the last confirmed slot. That human coordination step — operators agreeing on a restart height and booting together — is what the outside world saw as “Solana went down for several hours.”

   near-zero flat fees
        │  nothing prices demand; flooding is ~free
        ▼
   spam amplification            (bots send the same tx thousands of times)
        │  Gulf Stream forwards every copy; UDP can't throttle a source
        ▼
   forwarding storms             (one tx becomes many packets, cluster-wide)
        │  packets pile onto fetch + sig-verify
        ▼
   resource exhaustion           (CPU/memory/slot-time spent on spam)
        │  validators fall behind; no consensus on the tip in time
        ▼
   coordinated-restart HALT      (operators restart from last confirmed slot)
        └─ liveness failure, not safety — ledger intact, progress frozen

Every link is a direct consequence of a speed-first choice: free fees exist to serve users, forwarding exists to fill the pipeline, UDP exists to minimize latency, tight coupling exists to maximize throughput. Under normal load each is a feature. Under a flood, they compose into a halt.

The outages: dated liveness failures

The Outage Timeline grounded the mechanism in real, dated events. These figures are drawn from widely-reported post-mortems and news coverage; treat the exact durations as approximations, since public accounts differ on the minute-by-minute details.

• September 14, 2021 — the Grape Protocol IDO. The best-known incident. An IDO on the Grape Protocol drew a flood of bot transactions — reported at very high sustained packet rates — that overwhelmed the network. Validators fell out of consensus and the cluster was down for roughly 17 hours, recovered by a coordinated validator restart. This is the textbook instance of the full causal chain above: a hot event, spam amplification, forwarding storm, exhaustion, halt.
• 2022 — NFT-mint-driven halts and degradations. Through 2022 the network suffered further outages and severe slowdowns, several tied to candy-machine NFT mints where the same free-tx-flood pattern recurred, plus at least one incident traced to a specific bug in the transaction-processing path. Some were full halts requiring restarts; others were heavy degradation where the network limped rather than stopped.
• February 25, 2023 — a later degradation. A block-processing issue caused a significant slowdown and a restart, a reminder that not every incident was purely load-driven — some were software bugs in a system with little slack to absorb a bug’s effects.

The pattern across all of them is the same signature: loss of liveness under stress, ledger intact, recovery by human-coordinated restart. None was a break of the cryptography or the consensus rules. The chain was never wrong; it was, for hours at a time, stuck. That distinction is the single most important thing to carry out of the timeline, because it tells you exactly what needed fixing — not the safety layer, which held, but the layer that admits and prices load.

It actually happened — the September 2021 halt

On September 14, 2021, the Grape Protocol IDO on Solana triggered a transaction flood widely reported in the hundreds of thousands of packets per second aimed at the leaders. The network’s raw-UDP ingest and flat fees offered no way to throttle or price the surge; validators exhausted resources, forks proliferated faster than nodes could resolve them, and the cluster lost consensus. Recovery took a coordinated restart by validator operators and left the network offline for around 17 hours. The lesson the team drew is exactly the one this part is built on: with no admission control and no demand-based pricing, a single hot event can turn the network’s own forwarding into a self-inflicted denial-of-service. Every fix that followed targets one link in that chain.

The fixes: admission control and economic back-pressure

The last two content pages were the repair. The fixes split cleanly into two categories, and it is worth memorizing which link of the chain each one attacks, because that mapping is the understanding.

Category one — ingress admission control. These fixes attack the forwarding-storm → exhaustion link by giving the validator a way to tell packets apart and refuse the excess before it costs real work. Covered in The Ingress Fix — QUIC & Stake-Weighted QoS:

• QUIC replaces raw UDP. QUIC is a connection-oriented transport: a sender must establish a session, which gives the validator a handle on each source. That handle is the precondition for everything else — you cannot throttle a source you cannot identify. QUIC lets the validator cap connections per source, apply flow control, and drop a misbehaving peer instead of drowning in anonymous packets.
• Stake-weighted quality of service (QoS). Given that sources are now identifiable, the validator allocates ingress capacity in proportion to stake. A validator (or a connection sponsored by one) with 1% of stake is guaranteed roughly up to 1% of the leader’s ingest capacity. Spam from an unstaked or low-stake flood can no longer crowd out transactions arriving through well-staked paths, because the flooder has no stake to claim capacity with. This converts an anonymous free-for-all into a metered queue where the network’s own economic weight decides who gets heard.

Category two — economic back-pressure. These fixes attack the free-fees → spam-amplification link by making flooding cost something that scales with contention. Covered in Pricing the Load — Local Fee Markets & Priority Fees:

• Local fee markets. The key insight, first met at the hot-account ceiling, is that contention on Solana is measured per writable account. So the fee for priority is priced per account, not globally: a frenzy on one hot account raises the price of writing that account, without taxing an unrelated transfer elsewhere in the block. This is only possible because of the declared-footprint account model — the runtime already knows exactly which accounts a transaction contends for.
• Priority fees. On top of the base fee, a transaction can attach a priority fee — a per-compute-unit bid — that raises its position in the leader’s scheduling for the accounts it touches. Now, during a hot event, the way to land your transaction is to bid, not to spam. Volume stops being the cheapest lever; price becomes the lever. That is the whole point of a fee market: it converts a physical race (send more packets) into an economic auction (pay more), and an auction is something the system can price and shed, where a packet race is something it can only drown in.

   the causal chain            the fix that attacks that link
   ────────────────            ──────────────────────────────
   free flat fees          →   local fee markets + priority fees
    (spam is ~free)             (contention now costs, priced per account)

   spam → forwarding storm  →   QUIC (identify + throttle a source)
    (UDP can't throttle)        stake-weighted QoS (capacity ∝ stake)

   storm → exhaustion       →   admission control drops excess before
    (junk eats sig-verify)      it reaches the expensive stages

   exhaustion → halt        →   (no single fix — it is the composite result;
                                remove the earlier links and the halt
                                stops being reachable under normal load)

Read the mapping in one breath: QUIC and stake-weighted QoS are admission control — they decide whose packets even enter. Local fee markets and priority fees are economic back-pressure — they decide what it costs to be prioritized once inside. One is a bouncer at the door; the other is surge pricing at the bar. Together they reintroduce, as engineered mechanisms, the two things a mempool gave other chains for free: a buffer that throttles, and an auction that prices.

By the numbers — why bidding beats spamming

The economic fix works only if bidding is genuinely cheaper than flooding for a rational actor. Sketch the two strategies for landing one transaction during a hot event, with round numbers (illustrative, not live figures):

   goal: land 1 tx into a contested block

   spam strategy (old world, flat fee f):
     send N copies to raise odds; cost ≈ N · f
     with f tiny, N can be huge → you flood, and so does everyone
     → the storm is the equilibrium

   bid strategy (fee market, priority fee p):
     send 1 copy with a priority bid p that clears the account's market
     cost ≈ base + p, and p rises only with real contention
     → you pay for priority instead of manufacturing packets

The flat-fee world makes N (packet count) the free variable and f a fixed floor, so competition is a volume race the network must physically absorb. The fee-market world makes p (price) the free variable, so competition is an auction the network can account for and clear. The fix does not eliminate demand — nothing can — it changes the currency of competition from packets, which exhaust the leader, to lamports, which do not.

The throughline, landed

Put the whole arc together and the shape is unmistakable. Solana optimized one chain harder for throughput than anyone had tried before — and every optimization was a removal of slack. No mempool, synchronous execution, raw UDP, flat near-zero fees: each shaved latency, and together they left a system with almost nothing to absorb or price a surge. That was fine until the surges came, and when they did — Grape’s IDO, the 2022 mints, the later degradations — the missing slack showed up as liveness failures: the ledger stayed correct while progress froze, recovered only by operators restarting the cluster by hand.

The repair is the mirror image of the exposure. Every fix reintroduces, as deliberate engineering, a piece of the slack the original design threw away. QUIC and stake-weighted QoS put a throttleable, stake-aware buffer back at ingress, where the mempool’s admission control used to be. Local fee markets and priority fees put a demand-based auction back into pricing, where the mempool’s fee competition used to be — and, uniquely, priced it per account because the account model made per-account contention legible.

The single sentence to carry out of this part is the one the throughline has been circling from the start: you cannot optimize a global state machine that hard for throughput without trading away robustness to load — and if you do, you spend years engineering that robustness back, one shock absorber at a time. Solana’s outage history is not a story of a broken chain. It is a case study in the true cost of the hardware speed half of the book’s question, and the concrete price paid to earn the without falling apart half. The safety layer never bent; the load layer did, and had to be rebuilt. That is the whole part in one idea — and it is the honest counterweight to every earlier chapter that made the machine fast.