The Outage Timeline — What Actually Happened

The last three pages built the theory. Why a Speed-First Chain Can Halt argued that a single synchronous machine has less slack than a slow one; No Fee Market, and How Spam Amplifies showed why a near-zero, flat fee makes spam almost free; and Forwarding Storms & Resource Exhaustion traced how that free spam turns into duplicate-transaction floods that exhaust a validator’s CPU, memory, and network before a block is ever built. This page checks that theory against reality.

The plan is simple: walk the actual, dated outages of Solana mainnet-beta and, for each one, ask two questions. What was the trigger — and does it match the spam/exhaustion model? And what kind of failure was it — did the chain stop (liveness), or did it lie (safety)? The answers are consistent enough to be the whole point of the Part. Read every date and hour count below as a widely-reported approximation, not a stopwatch reading; the shape matters more than the second decimal place.

How to read an outage

Two distinctions do all the work here. Keep them separate as you read.

Liveness vs. safety. A liveness failure means the chain stops making progress — no new blocks, the whole network wedged, usually ending in a coordinated restart from an agreed-upon slot. A safety failure would be far worse: the chain keeps running but produces wrong answers — reversing a confirmed transaction, double-spending, or minting funds from nothing. Solana’s public incidents are overwhelmingly the first kind. That is not a small detail. A halted chain is embarrassing and costs uptime; a chain that silently corrupts balances is not money anymore.

Congestion vs. bug. Most — but not all — of the halts trace back to the load story from the previous pages: a flood of transactions the network could not shed cleanly. But at least one later outage was a plain software defect with nothing to do with spam. Lumping every outage under “Solana can’t handle load” is the wrong lesson; some were load, some were code.

   failure taxonomy
   ─────────────────────────────────────────────────────────
   liveness (halt)   →  chain stops, restart from known slot   ← all the big ones
   safety  (corrupt) →  chain lies: reversal / double-spend    ← no widely-reported case
   ─────────────────────────────────────────────────────────
   trigger: congestion  →  spam / bot flood exhausts resources ← 2021, spring 2022
   trigger: bug         →  a code defect wedges the network    ← Feb 2024 loader bug

September 14, 2021 — the Grape flood

This is the canonical one, and it maps cleanly onto everything the Part has taught.

It actually happened — the September 2021 halt (~17 hours)

On or around 14 September 2021, Solana mainnet-beta stopped producing blocks for roughly 17 hours. The widely-reported trigger was a flood of transactions from bots contending for a token sale — an IDO (initial DEX offering) on Grape Protocol. Reported peak inbound rates were extraordinary (figures on the order of hundreds of thousands to millions of transactions per second have been cited); treat the exact number as folklore, but the order of magnitude — far past what the network could process — is the point.

The flood exhausted validator resources and the network forked: validators could not converge on a single chain, and consensus stalled. There was no clean way to shed the excess load, so operators had no in-protocol recovery. The fix was social, not automatic: validator operators coordinated off-chain, agreed on a known-good slot, and performed a cluster restart from that slot with a patched client.

The lesson is exactly the throughline of this Part. A near-zero flat fee (see No Fee Market) meant the bots paid almost nothing to spam. Mempool-less forwarding meant that spam went straight at the leader with no global backpressure. And a single synchronous machine has no spare capacity to absorb a flood — so instead of degrading, it stopped. This was a liveness failure: the chain halted and was restarted. There is no widely-reported claim that the ledger was corrupted or that funds were minted or reversed during the event.

Notice how many earlier pages this single incident touches: the fee model, the forwarding path, the lack of a backpressure valve, and the restart-from-a-slot recovery. It is the worked example the whole Part is building toward.

Spring 2022 — the NFT-mint floods

The 2021 halt was not a one-off. Through spring 2022, the network saw further multi-hour outages and stretches of severe degradation, and the trigger rhymed with Grape: bot floods around popular mints, this time NFT mints rather than a token IDO.

It actually happened — the spring 2022 degradations (multi-hour)

Around late April into May 2022 (widely reported around 30 April – 1 May), and again in June 2022, Solana suffered major outages and heavy degradation. The commonly-cited cause was floods of transactions from bots racing to mint NFTs — enormous volumes of near-duplicate transactions the network could not process or shed cleanly. As in 2021, recovery involved coordinated validator action and, in the worst cases, a restart.

Same disease, new vector. An NFT mint is a scarce first-come-first-served event with no meaningful cost to retry, so bots submit the same transaction thousands of times, hoping one lands first. With no local fee market to price that contention, the only way to “win” was to out-spam everyone else, and the aggregate spam swamped the leader’s ingress. Again: liveness failures — the chain halted or crawled, but there is no widely-reported case of it corrupting balances.

Two structural facts made spring 2022 possible, and both are fixed by later pages in this Part:

• Ingress was undefended. Anyone could open a connection and fire transactions at the leader with no identity and no cost. The fix — QUIC & Stake-Weighted QoS — gives the leader a way to throttle and prioritize by stake instead of accepting all comers equally.
• Contention had no price. A hot mint account had no way to charge more under load. The fix — Local Fee Markets & Priority Fees — lets a contended account get expensive, so out-bidding replaces out-spamming.

February 2023 and February 2024 — not always congestion

The later two big incidents are important precisely because they break the pattern, and an honest timeline has to include them.

It actually happened — Feb 2023 (most of a day) and Feb 2024 (~5 hours)

In February 2023, mainnet-beta halted for the better part of a day (widely reported on the order of ~18–20 hours) and required a coordinated restart. The root cause was less cleanly a “spam flood” than the 2021/2022 events; reports pointed at issues in block propagation and consensus under load rather than a simple bot storm. Treat the specifics as contested and the duration as an approximation.

In February 2024, the network halted for roughly five hours. This one was traced to a bug in a legacy program loader — a software defect in how a particular class of programs was handled, not a congestion event at all. Recovery again meant a coordinated restart with a patched client.

The February 2024 outage is the counter-example that keeps the model honest. If your entire mental model is “Solana halts because spam exhausts it,” you will mis-diagnose 2024 — that halt was a plain bug in the loader, and no amount of fee-market or QoS work would have prevented it. The correct generalization is narrower and truer: a single synchronous global machine has little slack, so a wide class of triggers — floods or defects — can wedge the whole thing at once. Congestion was the most common trigger, not the only one.

Both were still liveness failures. In every one of these incidents, the recovery mechanism was the same in spirit as 2021: stop, agree on a known-good slot, restart. Nowhere in the public record is there a widely-reported case of the protocol confirming-then-reversing a transaction or minting funds during a halt.

The pattern, and the trend

Line the incidents up and two things jump out.

   date            approx duration   widely-reported trigger        failure class
   ─────────────   ───────────────   ────────────────────────────   ─────────────
   2021-09-14      ~17 hours         Grape IDO bot flood            liveness (halt)
   spring 2022     multi-hour(s)     NFT-mint bot floods            liveness (halt)
   2023-02         ~18-20 hours      propagation/consensus under    liveness (halt)
                                     load (contested specifics)
   2024-02         ~5 hours          legacy loader bug (not spam)   liveness (halt)
   ─────────────   ───────────────   ────────────────────────────   ─────────────
   (dates and durations are widely-reported approximations)

First, the failure class is stable: every major incident was a halt-and-restart, not a corruption. The speed-first bet cost uptime under stress, not the integrity of the money. That is the single most important sentence on this page, and it is why “Solana went down” is a genuinely different — and less severe — statement than “Solana lost funds.”

Second, the trend is toward fewer, shorter outages as the client matured. The fixes were not hand-waving; they were concrete engineering responses to the exact exhaustion mechanism from the previous page: QUIC-based ingress with stake-weighted quality-of-service to defend the leader’s front door, and local fee markets with priority fees to price contention so bidding replaces spamming. We study both next. State this trend carefully, though: reliability improved, it was not declared solved — treat any claim of a perfect uptime record with the same skepticism you’d apply to any dated performance number.

By the numbers — how “cheap to spam” becomes a halt

A back-of-envelope for why a flat, near-zero fee is structurally dangerous. Suppose a base fee is on the order of 5,000 lamports (≈ 0.000005 SOL) per signature, flat regardless of load:

   attacker budget          1 SOL  =  1,000,000,000 lamports
   cost per spam tx         ~5,000 lamports (flat, no surge)
   ─────────────────────────────────────────────────────────
   transactions bought      1e9 / 5e3  =  200,000 tx  for 1 SOL

   with duplicate forwarding (say each retried ~10×):
   effective load on leader ~2,000,000 tx-equivalents for 1 SOL

The exact lamport figure is illustrative and drifts with the client and SOL’s price — do not quote it as current. The durable point is the arithmetic: when the fee does not rise with load, a tiny budget buys an enormous, duplicated flood, and the leader has no price signal to shed it. A rising, per-account price (priority fees) changes that denominator so the same attack costs orders of magnitude more.

The architect’s lens

Reading the outage record is itself an architectural skill — the timeline is a diagnostic tool, not a list of embarrassments.

• Why does it exist? Because a design’s real failure modes only show up under adversarial production load. The timeline is the empirical record that tells you which of a chain’s theoretical weaknesses actually bit, and how often.
• What problem does it solve? It separates rumor from mechanism. Instead of “Solana is unreliable,” it lets you say precisely: these halts, from these triggers, of this failure class — and therefore these specific fixes.
• What are the trade-offs? The record is retrospective and hedged — dates, durations, and even root causes are widely-reported approximations, sometimes contested. It tells you what has broken, never guarantees what won’t.
• When should I avoid it? Do not use a dated incident to make a live claim (“Solana halts every few months”) — the trend changed. Use it to understand mechanisms, and hedge any current-state assertion with an absolute date.
• What breaks if I remove it? Without the timeline you’re left with either boosterism or FUD. You lose the ability to check theory against reality — which is exactly what turns “Solana can halt” from a slogan into an engineering diagnosis.

Check your understanding

1. What was the widely-reported trigger of the 14 September 2021 outage, roughly how long did it last, and how was the network recovered?
2. Explain the difference between a liveness failure and a safety failure. Which kind were Solana’s major outages, and why does the distinction matter for judging the trade-off?
3. The spring 2022 outages were driven by NFT-mint bot floods. Which two later fixes in this Part directly target that mechanism, and what does each one change?
4. Why is the February 2024 outage an important counter-example to the “spam always causes the halt” story? What is the more accurate generalization?
5. Why does every date and hour count on this page carry a hedge? Give one way you’d correctly cite this material versus one way you’d misuse it.

Show answers

1. A bot flood around the Grape Protocol IDO overwhelmed validators and forked the network; it lasted roughly 17 hours. Recovery was a coordinated cluster restart from a known-good slot with a patched client — a social, off-chain fix, since there was no in-protocol way to shed the load.
2. A liveness failure means the chain stops making progress (halts, then restarts); a safety failure means it keeps running but produces wrong results (reversals, double-spends, minting from nothing). Solana’s major outages were liveness failures. It matters because a halt costs uptime, whereas a safety failure would mean the ledger — the money itself — could no longer be trusted; the speed-first bet cost the former, not the latter.
3. QUIC & Stake-Weighted QoS defends the leader’s ingress by throttling and prioritizing connections by stake instead of accepting all comers equally, and Local Fee Markets & Priority Fees puts a rising price on contended accounts so participants out-bid rather than out-spam each other.
4. The February 2024 halt was traced to a bug in a legacy program loader, not a congestion event — so a spam-only model mis-diagnoses it, and fee/QoS fixes would not have prevented it. The accurate generalization is that a single synchronous global machine has little slack, so many triggers — floods or defects — can wedge the whole network at once; congestion was the most common trigger, not the only one.
5. Because the record is retrospective and the public figures are widely-reported approximations, sometimes contested, and reliability improved over time. Correct use: cite an incident to explain a mechanism (“a flat fee made spam nearly free, which enabled the 2021 flood”). Misuse: turn a dated incident into a live claim (“Solana halts every few months”) without an absolute date — the trend changed as the client matured.