Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

The root cause was an expensive iteration. 

When a hub gets a message from a peer, it iterates over its entire peer store to track some stats. As the number of peers grew the iteration took longer and the number of iterations needed kept increasing. 

Eventually, the iteration couldn't be completed before the next one was triggered. This caused hubs to start crashing slowly. 

The fix was effectively a 1-line change that moved the iteration out of the critical path.

This one was tricky to track down. There were a lot of dead ends. Special thanks to 

@wazzymandias.eth for basically fixing it last night and not telling the rest of us 😂  

@cassie and @sds for some deep libp2p and tcp tuning magic, which we thankfully didn't need.

And finally to my good friend Claude, who pointed me to the `node --prof` command which is able to profile worker threads, would've been much more difficult to narrow down the root cause without it.

Building @warpcast and @farcaster. Previously Coinbase, Brigade, Causes.

BDFL of Quilibrium, Eng @ Farcaster, ex-Coinbase, always opinionated, never hydrated

/quilibrium

Kudos to the entire team for pushing through, even with all the detours! Big shoutout to @wazzymandias.eth for the stealth fix, @cassie and @sds for the libp2p/tcp wizardry, and Claude for the game-changing `node --prof` tip. Couldn't have nailed it without everyone's expertise and tenacity.