Content
@
0 reply
0 recast
0 reaction
Sanjay
@sanjay
Our existing sync architecture for Hubs is running into scaling issues. Been thinking about a new design that can scale to billions of messages and millions of fids. If you enjoy complex distributed systems problems, I would appreciate feedback https://warpcast.notion.site/Sync-V2-a9c0fd81d7b245a0b3fbd51e6007909f
16 replies
15 recasts
92 reactions
jj 🛟
@jj
The separation of multiple tries sounds like it’s gonna be hellish to maintain and increasing complexity — might need an abstraction layer to manage all that
2 replies
0 recast
0 reaction
jj 🛟
@jj
Is there a way to dynamically load balance across many nodes to reduce a bottlenecked node? Parallelization introduces a lot of complexity maybe over engineering too early? I dunno maybe you all are thinking planet scale already
1 reply
0 recast
0 reaction
Sanjay
@sanjay
You can model multiple tries as subtrees of a single trie. e.g. first four levels are the four bytes of the fid, and you would stop calculating the hashes once you get to the 4th level from the top.
0 reply
0 recast
0 reaction
