Content pfp
Content
@
0 reply
0 recast
0 reaction

Varun Srinivasan pfp
Varun Srinivasan
@v
Scaling Gossip with Bundles Hubs are running into scaling issues with libp2p. We're proposing a change to "bundle" messages to fix some of these issues. This may add a ~1s delay to casts moving between clients. https://warpcast.notion.site/Scaling-Gossip-in-Hubble-e66c766fa6b04afcb407f4800134cd72?pvs=25
11 replies
11 recasts
183 reactions

jj πŸ›Ÿ pfp
jj πŸ›Ÿ
@jj
How did you determine the 1s delay? Are these dynamic batches or fixed in size?
2 replies
0 recast
1 reaction

jj πŸ›Ÿ pfp
jj πŸ›Ÿ
@jj
I get that batching will dramatically reduce the sizes, and I guess the dedup is fine.. do you have any data on how much each hub is dedupping? Perhaps you can get much more efficient with metadata only peering to more efficient pub IWANT in these batches
1 reply
0 recast
0 reaction

jj πŸ›Ÿ pfp
jj πŸ›Ÿ
@jj
Another thing might be to be clever with topics and batching. Lowest hanging fruit might be to differentiate historic sync vs realtime gossip if you’re not doing that already. So historic syncs you can have much larger batches vs gossip data might want to be smaller for latency
1 reply
0 recast
0 reaction