Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

The root cause was an expensive iteration. 

When a hub gets a message from a peer, it iterates over its entire peer store to track some stats. As the number of peers grew the iteration took longer and the number of iterations needed kept increasing. 

Eventually, the iteration couldn't be completed before the next one was triggered. This caused hubs to start crashing slowly. 

The fix was effectively a 1-line change that moved the iteration out of the critical path.

This one was tricky to track down. There were a lot of dead ends. Special thanks to 

@wazzymandias.eth for basically fixing it last night and not telling the rest of us 😂  

@cassie and @sds for some deep libp2p and tcp tuning magic, which we thankfully didn't need.

And finally to my good friend Claude, who pointed me to the `node --prof` command which is able to profile worker threads, would've been much more difficult to narrow down the root cause without it.

Building @warpcast and @farcaster. Previously Coinbase, Brigade, Causes.

BDFL of Quilibrium, Eng @ Farcaster, ex-Coinbase, always opinionated, never hydrated

/quilibrium

one person's "libp2p tuning magic" is another person's "story written in blood just trying to make libp2p function"

The system is not designed for this many peers. Gossip will likely break at our scale. I suspect the vast majority are airdrop farmers. We’re thinking about how to address this longer term.

It’s kinda crazy - I’m now seeing over 10k+ peers and still going

Awesome job team!

Thank you all for your work on this! 🙏

I have a friend named Claude too, such a helpful guy and never takes a break

Great stuff. Nothin like a one liner.
Thanks!

Grateful for the teamwork that eventually solved the issue. Special thanks to @wazzymandias.eth, @cassie, @sds, and Claude. Collaboration at its finest!

Node profiler can save a day sometimes. Node has its challenges but dev stack around v8 is just really good!

Kudos to the entire team for pushing through, even with all the detours! Big shoutout to @wazzymandias.eth for the stealth fix, @cassie and @sds for the libp2p/tcp wizardry, and Claude for the game-changing `node --prof` tip. Couldn't have nailed it without everyone's expertise and tenacity.