Skip to content
Solana · First Principles

Why a Speed-First Chain Can Halt

The part overview made a promise: Solana’s outages are not embarrassing bugs bolted onto an otherwise-robust design — they are the predictable shadow of the design. This page pays that promise off. It answers one question precisely: why is a chain optimized for throughput structurally more exposed to halting than a chain optimized for conservatism?

We build the answer from the book’s throughline — how do you build a single global state machine that runs at hardware speed without falling apart? The honest reading is that “at hardware speed” and “without falling apart” pull against each other, and Solana spent its early years discovering exactly where. Once you see why it can halt, the next three pages — the missing fee market, forwarding storms, and the outage timeline — read as consequences of a single set of choices rather than a list of unrelated incidents.

Recap: one synchronous chain, paid for with hardware

Section titled “Recap: one synchronous chain, paid for with hardware”

If you read the rest of this book, you can state Solana’s bet in one breath: maximize transactions-per-second-per-dollar on a single, synchronous layer-1, and pay for it with hardware and clever systems engineering. No sharding, no rollups, no second layer to hide latency behind — one machine-like chain that every validator runs in lockstep.

That bet has a cost that matters enormously for reliability: there is very little slack. When you tune any system to run near its hardware ceiling, you spend the headroom that a more conservative system keeps in reserve. Bitcoin deliberately caps itself at roughly seven transactions per second on hardware a Raspberry Pi can run — it is drowning in spare capacity by construction. Solana aims a validator’s 64 cores, 256+ GB of RAM, and multiple NVMe drives at the problem and tries to keep them busy. A busy system near its ceiling has nowhere to go when a flood arrives.

   headroom under load
   ───────────────────
   Bitcoin    [████████████████████░]  ~7 TPS on a Pi — vast unused margin
   Ethereum   [██████████████░░░░░░░]  L1 conservative; L2 absorbs the surge
   Solana     [███░░░░░░░░░░░░░░░░░░░]  tuned near the hardware ceiling
                    ▲ a flood here has little slack to absorb into

Slack is not waste — slack is what a system spends to stay live when the world misbehaves. Solana traded slack for speed. The rest of this page is about the specific mechanism that removes the slack.

The missing shock absorber: no mempool backpressure

Section titled “The missing shock absorber: no mempool backpressure”

Here is the single most important structural fact on this page. On Bitcoin and Ethereum, an unconfirmed transaction lands in a mempool — a shared, per-node holding pool of pending transactions. The mempool is not just a waiting room; it is a shock absorber and a pressure gauge.

How a mempool sheds load

Section titled “How a mempool sheds load”

When a Bitcoin or Ethereum node’s mempool fills up, three things happen automatically, with no coordination:

  1. It backs pressure onto the sender. A full mempool means low-fee transactions simply won’t get in — nodes evict the cheapest. The network is telling senders “pay more or wait,” and that signal is visible before a transaction is mined.
  2. It sheds excess load locally. Each node independently drops transactions it can’t hold. The flood is absorbed and thinned at the edges instead of hitting the block producer all at once.
  3. It decouples arrival from execution. Transactions arrive continuously but are only executed when mined into a block, roughly every ten minutes (Bitcoin) or twelve seconds (Ethereum). The pool is a buffer between a spiky arrival rate and a steady execution rate.
   Bitcoin / Ethereum
   ──────────────────
   senders ─► mempool (buffer + gauge) ─► miner picks top-fee txs ─► block
                  │
                  └─ full? evict cheapest, back-pressure senders  ◄── slack lives here

A mempool is a queue with an eviction policy, and a queue with an eviction policy is exactly how you survive more demand than you can serve: you serve what you can and drop the rest cleanly, before it costs you anything.

Gulf Stream: a mempool-less firehose

Section titled “Gulf Stream: a mempool-less firehose”

Solana threw the mempool out. Its transaction-forwarding protocol, Gulf Stream, has clients and validators forward transactions directly to the expected upcoming leader — the validator scheduled to produce the next few blocks — rather than gossiping them into a shared pending pool that every node holds. Because Solana’s Proof of History clock makes the leader schedule known in advance, a sender can push a transaction straight at the node that will build the block.

This is a genuinely clever throughput optimization. It removes a whole gossip round-trip and lets the leader start working on transactions before its slot even begins — part of why Solana can pipeline at the speed it does. But look at what it also removes:

   Solana
   ──────
   senders ─► (no shared pool) ─► forward straight at the expected leader
                                        │
                                        └─ leader's ingress must absorb the whole
                                           firehose itself — no edge to shed at

There is no shared pending pool, so there is no shock absorber and no pressure gauge. Excess load is not thinned at thousands of independent mempools; it is aimed, as a firehose, at a handful of upcoming leaders. And crucially, in Solana’s early design there was no fee-based eviction at that ingress — one transaction was as cheap and as welcome as another, so there was no clean way to say “drop the low-value flood, keep the real traffic.” That combination is the missing shock absorber. We spend the whole next page on why the absent fee market turns a flood into an amplified flood.

The architect’s lens

Section titled “The architect’s lens”

Gulf Stream — mempool-less forwarding — is the major design choice this page turns on, so it earns the lens.

  • Why does it exist? To hit hardware-speed throughput, Solana wanted the leader working on transactions before its slot starts and wanted to avoid a network-wide gossip pool as a bottleneck. Forwarding straight to the known upcoming leader removes a round-trip and a shared data structure from the hot path.
  • What problem does it solve? The latency and coordination cost of a global mempool. A mempool every node must hold, sync, and reconcile is overhead you don’t want when you are trying to run a chain like a CPU pipeline at maximum utilization.
  • What are the trade-offs? You delete the mempool’s other jobs: buffering spiky load, shedding excess cleanly at the edges, and signalling congestion back to senders through fee-based eviction. The firehose now lands on the leader with no queue to absorb it.
  • When should I avoid it? Any design that must degrade gracefully under adversarial overload should keep a backpressure mechanism. A mempool-less firehose optimizes throughput on the happy path at the cost of the unhappy path. If your failure budget is tight and floods are expected, keep the shock absorber.
  • What breaks if I remove it? Remove Gulf Stream and you’d reintroduce a mempool: more robust under load, but slower and with a global pool back in the pipeline. That is the exact tension the throughput bet was trying to escape — you can’t have the raw speed and the free shock absorber at the same time.

Liveness vs safety — say it precisely

Section titled “Liveness vs safety — say it precisely”

Distributed-systems people split correctness into two properties, and the whole judgement of Solana’s outages hinges on which one failed.

  • Safety — “nothing bad ever happens.” The ledger never confirms a transaction and later reverses it; funds are never minted from nothing; two validators never finalize conflicting histories. Safety is about correctness of the results the chain produces.
  • Liveness — “something good eventually happens.” The chain keeps making progress — it keeps producing and finalizing new blocks. Liveness is about the chain not getting stuck.
   safety   = the answers it gives are never wrong        (integrity)
   liveness = it keeps giving answers                      (progress)


   Solana's outages:  liveness FAILED  (it stopped)
                      safety   HELD     (it never lied about the ledger)

Solana’s outages were overwhelmingly liveness failures: the network stopped producing blocks and had to be restarted. There is no widely-reported case of the protocol confirming-then-reversing a transaction or minting funds during these events — safety held. That distinction is not a detail; it is the entire difference between “the ATM was down for a few hours” and “the bank silently changed your balance.”

Why restarting isn’t automatic — off-chain coordination

Section titled “Why restarting isn’t automatic — off-chain coordination”

Here is the part that surprises people. When Solana halts, it does not self-heal. A conservative chain like Bitcoin has no concept of “halting and restarting” — nodes just keep gossiping and the longest valid chain eventually wins; the protocol grinds forward on its own. Solana’s restart is a human, off-chain, social process:

   1. the network stops finalizing new blocks (liveness lost)
   2. validator operators coordinate off-chain — Discord, calls, forums
   3. they agree on a last-known-good slot to restart from
   4. enough operators (by stake) restart their nodes from that slot
   5. the network resumes producing blocks

Why can’t it just recover itself? Because Solana’s consensus (Tower BFT, a PoH-anchored BFT) needs a supermajority of stake voting on a shared view to finalize. When a flood fractures that view — validators fall behind, fork, and can’t agree on the tip — there is no automatic path back to a single agreed history; the safe move is to stop rather than risk a safety violation. Recovery then requires operators to manually converge on a restart point. The chain chose to halt (protect safety) rather than guess (risk safety). That is the correct trade — but it means liveness recovery depends on people coordinating, not on the protocol healing itself.

A halt is a less severe failure than a reversal

Section titled “A halt is a less severe failure than a reversal”

Put the two together and you get the fair verdict on Solana’s reliability record. Halting is a fundamentally different — and less severe — failure mode than confirming-then-reversing a transaction.

  • A halt freezes the ledger. Nothing new happens, but nothing already-true becomes false. When it restarts, your confirmed balance is exactly what it was. You lost access, for a while.
  • A reversal rewrites the ledger. A transaction you watched confirm — a payment you accepted, a trade you settled — is undone. You lost the thing itself.

An honest reliability review says both true things at once: Solana has halted more than Bitcoin or Ethereum, and those halts have been liveness failures that left the money intact. It bought throughput by trading away robustness-to-load — uptime under a flood — not by trading away the integrity of the ledger. For a payments-grade system that is the right corner to be weak in, but it is a real weakness, and the following pages are about how the team spent years buying the uptime back.

Setting up the root causes

Section titled “Setting up the root causes”

Everything on the next few pages is a consequence of what you just read. The same choices that make Solana fast are the ones that make it fragile under a flood:

   design choice (for speed)          reliability consequence (under a flood)
   ─────────────────────────────      ──────────────────────────────────────
   mempool-less Gulf Stream        ►  no shared pool to buffer or shed load
   near-zero, flat fees            ►  no price signal to drop low-value spam  → page 3
   forward straight to leader      ►  retries/duplicates flood ingress        → page 4
   one synchronous chain           ►  little slack; the flood hits everything → page 5

Then the story turns constructive. The ingress fix (QUIC and stake-weighted quality-of-service) and pricing the load (local fee markets and priority fees) are exactly the shock absorbers this page says were missing — retrofitted onto a firehose. Read forward:

→ Next: No Fee Market, and How Spam Amplifies · Back to the part overview

Check your understanding

Section titled “Check your understanding”
  1. Why does optimizing a single synchronous chain for throughput leave it with “little slack,” and why does slack matter for staying live under load?
  2. What three jobs does a Bitcoin/Ethereum mempool do under load, and which of them does Gulf Stream’s mempool-less forwarding remove?
  3. Define liveness and safety precisely. Which one did Solana’s historical outages violate, and which one held?
  4. Why can’t Solana automatically recover from a halt the way Bitcoin grinds forward on its own? What has to happen instead?
  5. Explain why a halt-and-restart is a less severe failure mode than a confirmed-then-reversed transaction, in terms of what a user actually loses.
Show answers
  1. Tuning a system to run near its hardware ceiling spends the headroom a conservative system keeps in reserve. Slack is the spare capacity a system uses to keep making progress when demand spikes or the world misbehaves; with little slack, a flood has nowhere to be absorbed and can tip the chain over instead of merely slowing it.
  2. A mempool (a) backs pressure onto senders (a full pool means pay-more-or-wait, a visible congestion signal), (b) sheds excess load locally by evicting the cheapest transactions at each node’s edge, and (c) decouples spiky arrival from steady execution as a buffer. Gulf Stream removes all three: there is no shared pending pool, so excess is aimed at the upcoming leader as a firehose with no edge to shed at.
  3. Safety = “nothing bad ever happens” — the ledger never confirms-then-reverses, never mints from nothing, never finalizes conflicting histories (correctness of results). Liveness = “something good eventually happens” — the chain keeps producing and finalizing blocks (progress). Solana’s outages violated liveness (it stopped); safety held (the ledger was never corrupted).
  4. Because Tower BFT needs a stake supermajority voting on a shared view to finalize; when a flood fractures that view (validators fall behind and fork), there’s no automatic path back to one agreed tip, and stopping is the safe move rather than guessing and risking a safety violation. Recovery then requires validator operators to coordinate off-chain on a last-known-good slot and restart their nodes from it. Bitcoin, by contrast, just keeps gossiping and lets the longest valid chain win.
  5. A halt freezes the ledger: nothing new happens, but nothing already-true becomes false — when it restarts, confirmed balances are unchanged, so the user lost only access/time. A reversal rewrites the ledger: a transaction the user saw confirm is undone, so the user lost the settled thing itself (money). Frozen-but-intact is far less severe than settled-then-rewritten.