Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

The root cause was an expensive iteration. 

When a hub gets a message from a peer, it iterates over its entire peer store to track some stats. As the number of peers grew the iteration took longer and the number of iterations needed kept increasing. 

Eventually, the iteration couldn't be completed before the next one was triggered. This caused hubs to start crashing slowly. 

The fix was effectively a 1-line change that moved the iteration out of the critical path.

This one was tricky to track down. There were a lot of dead ends. Special thanks to 

@wazzymandias.eth for basically fixing it last night and not telling the rest of us 😂  

@cassie and @sds for some deep libp2p and tcp tuning magic, which we thankfully didn't need.

And finally to my good friend Claude, who pointed me to the `node --prof` command which is able to profile worker threads, would've been much more difficult to narrow down the root cause without it.

Building @warpcast and @farcaster. Previously Coinbase, Brigade, Causes.

BDFL of Quilibrium, Eng @ Farcaster, ex-Coinbase, always opinionated, never hydrated

/quilibrium

It’s kinda crazy - I’m now seeing over 10k+ peers and still going

Awesome job team!

@sanjay just finished a scan and it’s 16k+ peers…

That’s insane. Are you counting unique peer ids? Via getCurrentPeers?

Don’t know if such a thing exists (or is possible to find out either from hub data or maybe FC team) but I’d be fascinated to see a graph of number of hubs over time (I.e. when did the spike start). Am assuming it’s a proper ‘vertical line farmer’ spike too