Skip to content
Solana · First Principles

Forwarding Storms & Resource Exhaustion

The previous page established the fuel: with near-zero fees and no price that rises under load, a bot can send tens of thousands of transactions a second for the cost of a rounding error. This page follows that fuel into the fire. We trace the exact causal chain by which a flood of cheap transactions becomes a halted network — not by attacking any one cryptographic primitive, but by doing something far more mundane: sending more work than the machines can hold in memory and chew through in time.

The throughline of this whole book is how do you build a single global state machine that runs at hardware speed without falling apart? This page is the “falling apart.” It is the story of what happens when the very design choices that make Solana fast — forwarding transactions ahead to the next leader, a firehose UDP ingress with no gate, unbounded in-memory queues — meet an adversary who is happy to hand you infinite work. Everything the Reliability overview promised to explain converges here.

The mechanism, in one sentence

Section titled “The mechanism, in one sentence”

A leader that cannot process transactions as fast as they arrive builds an unbounded backlog; the backlog consumes memory and CPU until the validator can no longer keep pace with slot timing, and the network diverges.

Every clause in that sentence is a link in a chain. Break any one and the halt does not happen. That is exactly why the fixes on the next pages work — they each snap a different link. Let’s build the chain from first principles.

Gulf Stream: forwarding is a feature until it’s a weapon

Section titled “Gulf Stream: forwarding is a feature until it’s a weapon”

Most blockchains have a mempool: a shared pool of pending transactions that every node holds and gossips. Solana deliberately does not. Because Proof of History gives every validator a known leader schedule — who produces blocks in which future slots is public in advance — a client (or an RPC node) can send a transaction directly to the validator that is about to be leader, instead of broadcasting it to everyone and waiting. This is Gulf Stream, Solana’s “mempool-less” forwarding.

In the happy path this is brilliant. It removes the global mempool’s memory cost and lets the leader start ingesting a transaction before its slot even begins, so the leader’s pipeline is already warm when its turn arrives. No wasted round-trips, no duplicated gossip storm.

Now feed it an adversary. The same property — transactions flow toward the upcoming leaders — means a flood does not spread out and thin; it concentrates. Every spam transaction is aimed at the small set of validators scheduled to lead soon, and forwarded onward as the schedule advances. What was designed as a smart routing optimization becomes a forwarding storm: the same load, duplicated and re-forwarded across nodes, all funneling into whichever validator is about to hold the pen.

   NORMAL (Gulf Stream as intended)
   client ─► forwards to next leader ─► leader ingests early ─► warm pipeline ✔


   UNDER FLOOD (forwarding storm)
        bots ─┐
        bots ─┼─► ALL forwarded toward the SAME upcoming leaders
        bots ─┤        │
        bots ─┘        ▼
              re-forwarded as schedule rolls over ──► load never disperses,
                                                      it accumulates on the leader

The key insight: forwarding does not create back-pressure. A node that receives more than it can handle does not tell senders “slow down” — instead it forwards or queues, and the problem moves downstream to the next leader intact. The storm rides the schedule forward instead of dying out.

Resource exhaustion: where the work actually piles up

Section titled “Resource exhaustion: where the work actually piles up”

A validator is not magic; it is a program on a machine with finite RAM and a finite number of cores. Trace a single transaction through the leader’s ingest path and you find three places the flood turns into resource pressure.

1. Unbounded queues eat memory

Section titled “1. Unbounded queues eat memory”

Transactions arrive faster than they can be processed, so they wait in an in-memory buffer. If that buffer has no hard cap, its size is dictated by the sender, not the operator. A sustained flood grows the queue without bound, and the validator’s memory footprint climbs until the OS starts to thrash or the process is killed. This is the first and most direct failure: the queue is a bucket with the tap fully open and no overflow.

   arrival rate  >  processing rate     ⇒   queue depth grows without bound
   ───────────────────────────────────      ────────────────────────────────
   (flood: 100k tx/s)  (leader: N/s)          RAM ↑ ↑ ↑  →  thrash / OOM kill

2. Signature verification and scheduling saturate CPU

Section titled “2. Signature verification and scheduling saturate CPU”

Before a transaction can be executed, its signatures must be verified — an Ed25519 check per signature (see signing and verification). That is real, non-trivial CPU work, and it happens whether or not the transaction is valid, useful, or ever lands in a block. A spam transaction that will be dropped still costs a signature verification to discover that.

Then the Sealevel scheduler must sort transactions into conflict-free batches — the very layering you can watch happen in solmini’s schedule. Scheduling is itself work, and it competes for the same cores that execution and PoH hashing need. Under a flood, the CPU spends its cycles on deciding what to do with garbage instead of producing a block.

// From solmini's runtime: BEFORE a batch can even be scheduled, every tx pays
// its compute cost. `simulate_compute` stands in for the real per-tx work —
// signature verification + program execution — that a leader cannot skip.
fn run_tx(
    programs: &HashMap<Pubkey, Box<dyn Program>>,
    compute_per_tx: u64,
    tx: &Transaction,
    accounts: &mut [Account],
) -> Result<()> {
    simulate_compute(tx, compute_per_tx); // ← non-vote spam pays this too
    match programs.get(&tx.program) {
        Some(p) => p.process(accounts, &tx.data),
        None => Err(SolError::UnknownProgram(tx.program.short())),
    }
}

The toy runtime lets you raise compute_per_tx and watch throughput collapse — a controlled version of what a flood does to a real leader for free. Every extra transaction in the queue is another simulate_compute the machine must pay before it can make any forward progress.

3. The leader falls behind slot timing

Section titled “3. The leader falls behind slot timing”

Here is the clause that turns pressure into a halt. A Solana slot is roughly 400 ms (see Proof of History); the leader must produce and start propagating a block inside that window or it forfeits the slot. If memory pressure and CPU saturation slow the leader down, it cannot finish its block on time. It either produces a late, partial block or produces nothing. Either way, the network’s carefully clocked cadence breaks — and a chain whose whole design assumes the clock is honored does not degrade gracefully when it isn’t.

From exhaustion to divergence to halt

Section titled “From exhaustion to divergence to halt”

A slow, overloaded leader is not yet a dead network. The lethal step is what happens across many validators at once. When leaders are dropping or delaying blocks and validators are starved of CPU and memory, they stop seeing a single consistent view of recent slots. Some validators vote on one fork of recent history; others, having received different or late data, vote on another. The network diverges — it forks — and no fork gathers the supermajority of stake needed to move optimistic confirmation forward.

   healthy:   all validators converge on one chain of slots ──►──►──►
   under load:
              validator A ─► slot n on fork X
              validator B ─► slot n on fork Y   (got late/partial data)
              validator C ─► starved, no vote
                     │
                     ▼
              no fork reaches supermajority  ⇒  no finalization  ⇒  progress stops

Once no fork can gather a supermajority, the chain stops finalizing. That is a halt: the machine is alive but produces no agreed-upon new state. And because the validators cannot agree their way out of it while still drowning in load, the only reliable recovery is out-of-band: operators coordinate — on social channels, not on-chain — to restart from a known-good slot, discarding the divergent tail and booting a fresh cluster from a shared snapshot. We walk through a real instance of exactly this on the outage timeline.

Note the character of this failure. It is a liveness failure — the chain stops — not a safety failure. No transaction was reversed after finalization, no funds were minted from nothing. The ledger’s integrity survived; its availability did not. That distinction is the whole reason the trade-off was judged survivable enough to fix rather than abandon.

Under the hood — why UDP ingress had no brakes

Section titled “Under the hood — why UDP ingress had no brakes”

The deepest link in the chain is the front door. In the era of the early outages, a validator’s transaction ingress was UDP-based, and UDP is connectionless: there is no handshake, no session, no notion of who is on the other end. A packet either arrives or it doesn’t. That has one fatal consequence for load control.

  • No admission control. With no connection to accept or refuse, a validator could not decline to talk to a spammer. Any sender, anywhere, could push packets at any validator with no cost of establishing a relationship first — the network equivalent of a door with no lock and no bouncer.
  • No identity to rate-limit against. UDP source addresses are trivially spoofable and carry no stake or reputation. The validator had no reliable way to say “this sender is flooding me, throttle them specifically” — because it could not reliably tell senders apart at all.
  • No back-pressure to the source. TCP-style flow control lets a receiver signal “slow down.” Raw UDP ingest gives the receiver no such lever; it can only drop packets locally, after they’ve already cost bandwidth and buffer space to receive.

So the ingress had exactly the wrong shape for adversarial load: it accepted unlimited work from unlimited, unidentifiable senders, with no way to charge, refuse, or slow any of them. This is precisely the hole the QUIC and stake-weighted QoS fix was built to close — QUIC adds connections (so there is a peer to accept, refuse, and rate-limit), and stake weighting gives that peer an identity worth trusting.

The architect’s lens

Section titled “The architect’s lens”

Gulf Stream forwarding is a genuine architectural component, so it earns the five questions — this time read through the reliability lens, because its trade-offs are the whole subject of this page.

  • Why does it exist? To let a leader ingest transactions before its slot begins by exploiting the publicly-known leader schedule, so the pipeline is warm and there is no global mempool to hold and gossip. It is a throughput optimization, born of Solana’s bet to scale a single fast L1 rather than scale out.
  • What problem does it solve? The memory and latency cost of a shared mempool: no node has to hold every pending transaction, and no transaction wastes a broadcast to nodes that will never lead soon.
  • What are the trade-offs? Forwarding concentrates load on the upcoming leaders instead of dispersing it, and by itself carries no admission control or back-pressure — so under a flood it amplifies rather than absorbs. The optimization that saves memory in the happy path is the amplifier in the adversarial one.
  • When should I avoid it? You never turn Gulf Stream off, but you must never deploy it naked — it needs a gated ingress (QUIC), an identity to prioritize by (stake weighting), and a price that rises under load (local fee markets) in front of it. Forwarding without those is a firehose with no valve.
  • What breaks if I remove it? Remove the guards and you get exactly the 2021 failure — a concentrated, unbounded flood exhausts leaders and halts the chain. Remove Gulf Stream itself and you reintroduce a global mempool’s memory and gossip cost, giving up the warm-pipeline latency win that is part of why Solana is fast at all.

Check your understanding

Section titled “Check your understanding”
  1. Solana has no global mempool. What does Gulf Stream do instead, and why does that same design concentrate rather than disperse a spam flood?
  2. Name the three distinct resources a forwarding storm exhausts, and say specifically what work consumes each one — including work spent on transactions that never land in a block.
  3. Walk the full causal chain from “cheap spam” to “manual restart,” naming each link. Which single link turns slowness into a halt?
  4. Why was the era’s UDP-based ingress unable to protect the leader? Give the three properties it lacked and tie each to a consequence.
  5. The 2021 outage was a liveness failure, not a safety failure. Explain the difference and why it matters for judging whether the throughput bet was worth fixing rather than abandoning.
Show answers
  1. Because the leader schedule is public in advance, clients send (and nodes forward) transactions directly toward the validators about to be leader, so no global mempool is needed and the leader ingests early. That very property means a flood is all aimed at the same small set of upcoming leaders and re-forwarded as the schedule rolls forward, so the load funnels onto the leader rather than thinning out across the network.
  2. Memory — unbounded in-memory queues grow without a hard cap because the sender, not the operator, sets the arrival rate. CPU (signature verification) — every transaction costs an Ed25519 check to process, paid even for spam that will be dropped. CPU (scheduling) — the Sealevel scheduler must sort arrivals into conflict-free batches, competing with execution and PoH hashing for the same cores.
  3. Cheap spam (no fee market) → forwarded and concentrated on upcoming leaders (Gulf Stream) → unbounded queues grow (memory) → signature-verify and scheduling saturate the CPU → the leader misses its ~400 ms slot timing → validators receive late/partial data and diverge (fork) → no fork reaches a supermajority, so finalization stops (halt) → operators coordinate a restart from a known slot. The leader falling behind slot timing is the link that converts slowness into a halt.
  4. UDP is connectionless, so the ingress had: no admission control (no connection to accept or refuse, so it could not decline a spammer); no identity to rate-limit against (spoofable source addresses with no stake or reputation, so no way to throttle a specific flooder); and no back-pressure (no way to signal “slow down” to the source — it could only drop packets locally after already paying to receive them).
  5. A liveness failure means the chain stopped making progress and had to be restarted; a safety failure would mean it produced incorrect results, like reversing a finalized transaction or minting funds. In 2021 the ledger stayed correct — only availability was lost — so the bet cost uptime under load, not the integrity of the money. That is a far less severe and far more fixable failure mode, which is why the response was to gate the ingress and price the load rather than redesign the chain.