Solana Network Avoids Catastrophic Failure by a Hair's Breadth

A recent incident on Solana's network nearly caused the loss of finality due to a routing bug at hosting provider Teraswitch. The bug, which occurred early Wednesday morning, took almost 29% of staked SOL offline, bringing the chain within 4.51 percentage points of losing its ability to finalize transactions.

The Solana Foundation's delegation program has a limit on how much stake can be held by a single autonomous system at 25%. However, AS20326 holds some 27% of staked SOL and experienced a complete outage, affecting 94% of its holdings. This was not an isolated incident as further 14.1 million SOL went offline across latitude.sh, Limestone, Butterfly Research, and Allnodes.

The bug occurred when a default route from Teraswitch's Miami site was propagated with its metric and communities stripped. This caused edge routers to read it as locally originated and preferred it over their real route, leading to a chain reaction that left some 12 sites in London, Amsterdam, Dublin, Frankfurt, Singapore, and Tokyo without valid routes.

Fortunately, engineers at Teraswitch quickly identified the issue and resolved it within ten minutes. The service returned by 04:16:15 UTC. However, this incident highlights the importance of having robust routing systems to prevent such incidents in the future.