Farcaster Devs

fc-updates

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

The root cause was an expensive iteration. 

When a hub gets a message from a peer, it iterates over its entire peer store to track some stats. As the number of peers grew the iteration took longer and the number of iterations needed kept increasing. 

Eventually, the iteration couldn't be completed before the next one was triggered. This caused hubs to start crashing slowly. 

The fix was effectively a 1-line change that moved the iteration out of the critical path.

This one was tricky to track down. There were a lot of dead ends. Special thanks to 

@wazzymandias.eth for basically fixing it last night and not telling the rest of us 😂  

@cassie and @sds for some deep libp2p and tcp tuning magic, which we thankfully didn't need.

And finally to my good friend Claude, who pointed me to the `node --prof` command which is able to profile worker threads, would've been much more difficult to narrow down the root cause without it.

Building @warpcast and @farcaster. Previously Coinbase, Brigade, Causes.

BDFL of Quilibrium, Eng @ Farcaster, ex-Coinbase, always opinionated, never hydrated

/quilibrium

It’s kinda crazy - I’m now seeing over 10k+ peers and still going

Awesome job team!

The system is not designed for this many peers. Gossip will likely break at our scale. I suspect the vast majority are airdrop farmers. We’re thinking about how to address this longer term.

Might need to designate a few supernodes (but it’ll make things less decentralized) 

Or need to limit the number of nodes that can forward messages…

There are also some probabilistic dissemination things you can do too..

All solvable problems with the best minds at work! Believe in you folks!