Sanjay on Warpcast

Content pfp

0 reply

0 recast

0 reaction

Sanjay pfp

Our existing sync architecture for Hubs is running into scaling issues. Been thinking about a new design that can scale to billions of messages and millions of fids. If you enjoy complex distributed systems problems, I would appreciate feedback https://warpcast.notion.site/Sync-V2-a9c0fd81d7b245a0b3fbd51e6007909f

16 replies

15 recasts

92 reactions

vrypan |--o--| pfp

Knowing very little about the internals of Hubble, I'm at risk of saying something totally useless, but have you checked Prolly Trees? I attended this presentation a couple of months ago, and I feel that it could be related to optimized hub sync. https://youtu.be/X8nAdx1G-Cs?si=1g5RdIKXg1S0mVA2&t=340

2 replies

0 recast

19 reactions

woj pfp

thanks for doing this in public

0 reply

1 recast

11 reactions

Manan pfp

I couldn't comment on the notion. >The hubs will only hold the last 14 days worth of data in the trie, data beyond that will be pruned daily. Does this mean that any gossip holes not filled within 14 days will be left open until a catchup sync finds it is lagging by more than 10% of the messages?

1 reply

0 recast

0 reaction

MJC pfp

what would you say are the downstream effects for app developers, if any?

1 reply

0 recast

0 reaction

jj 🛟 pfp

Lastly super basic batching, compression, dedup might offer some serious roi in storage, latency, and performance

1 reply

0 recast

0 reaction

jj 🛟 pfp

The separation of multiple tries sounds like it’s gonna be hellish to maintain and increasing complexity — might need an abstraction layer to manage all that

2 replies

0 recast

0 reaction

jj 🛟 pfp

Global state trie effectiveness hinges on its ability to rapidly propagate changes to all nodes — which hinges on the broadcasting mechanism which needs to fast/efficient

1 reply

0 recast

0 reaction

jj 🛟 pfp

For snapshotting it’ll be really important to add integrity checks at scale this becomes a bigger and bigger problem where you might need to think about error handling or partially corrupt snapshots

1 reply

0 recast

0 reaction

Wasif Iqbal pfp

@wazzymandias.eth

https://datatracker.ietf.org/doc/html/rfc3284

0 reply

0 recast

0 reaction

xh3b4sd ↑ pfp

Not knowing anything about hubs, try to * write data the way you want to read it * use hash rings for data locations * vertically scale for separate concerns * prioritize what's more important * checkpoint progress of processes * reduce complexity of data models * stream data chunks instead of polling Happy to chat!

0 reply

0 recast

0 reaction

frederick pfp

I'll be watching this...

0 reply

0 recast

0 reaction

Chase pfp

"Considered a bittorrent like approach to syncing fid specific messages, but it’s hard to determine which hubs have the right messages for the fid." Can you say more about this?

1 reply

0 recast

0 reaction

Kyle Tut pfp

0 reply

0 recast

0 reaction

Kaushal pfp

@fryorcraken.eth and the @waku team are the go to peeps.

0 reply

0 recast

3 reactions

aaalex.eth 🎩 pfp

aaalex.eth 🎩

I'm loving the transparency

0 reply

0 recast

1 reaction

ukstv pfp

Probably irrelevant based on the doc, and it’s not really my field: is there a place here for some sort of set reconciliation protocol? (Feel free to ignore, if its irrelevant)

0 reply

0 recast

0 reaction