Content
@
0 reply
0 recast
0 reaction
Sanjay
@sanjay
Our existing sync architecture for Hubs is running into scaling issues. Been thinking about a new design that can scale to billions of messages and millions of fids. If you enjoy complex distributed systems problems, I would appreciate feedback https://warpcast.notion.site/Sync-V2-a9c0fd81d7b245a0b3fbd51e6007909f
16 replies
15 recasts
92 reactions
jj 🛟
@jj
The separation of multiple tries sounds like it’s gonna be hellish to maintain and increasing complexity — might need an abstraction layer to manage all that
2 replies
0 recast
0 reaction
jj 🛟
@jj
Is there a way to dynamically load balance across many nodes to reduce a bottlenecked node? Parallelization introduces a lot of complexity maybe over engineering too early? I dunno maybe you all are thinking planet scale already
1 reply
0 recast
0 reaction
Sanjay
@sanjay
Not planetscale, but the way we're doing it now is extremely inefficient by only doing pairwise syncs every 2 hours. So, something between 1-1 syncs to 1-n, but not every peer syncs with every other peer all the time.
0 reply
0 recast
0 reaction
