Farcaster Devs

Our existing sync architecture for Hubs is running into scaling issues. Been thinking about a new design that can scale to billions of messages and millions of fids. If you enjoy complex distributed systems problems, I would appreciate feedback

Our existing sync architecture for Hubs is running into scaling issues. Been thinking about a new design that can scale to billions of messages and millions of fids. If you enjoy complex distributed systems problems, I would appreciate feedback 

https://warpcast.notion.site/Sync-V2-a9c0fd81d7b245a0b3fbd51e6007909f

Knowing very little about the internals of Hubble, I'm at risk of saying something totally useless, but have you checked Prolly Trees? I attended this presentation a couple of months ago, and I feel that it could be related to optimized hub sync.

Knowing very little about the internals of Hubble, I'm at risk of saying something totally useless, but have you checked Prolly Trees? I attended this presentation a couple of months ago, and I feel that it could be related to optimized hub sync. https://youtu.be/X8nAdx1G-Cs?si=1g5RdIKXg1S0mVA2&t=340

Dad, geek, conversation liquidity provider. vrypan.net (home), /weatherxm (work), @l--o--l (fun). Also https://paragraph.xyz/@purplesubmarine

This is very interesting, thank you. I had looked into other merkle tree alternatives like Merkle Mountain Ranges and Verkle Trees, but key difference with hubs is we also remove messages which makes rebuilding them costly. Prolly trees seem promising, will investigate deeper.

Thanks for the heads up on this talk @vrypan.eth - super interesting.