Content
@
0 reply
20 recasts
20 reactions
Varun Srinivasan
@v
One point of feedback we got on the Snapchain design doc is that we didn't fully explain why sync in the current model is a problem. @sanjay wrote up a good thread on our internal slack, which i'll try to summarize below. https://warpcast.notion.site/Snapchain-v2-Public-10e6a6c0c101807aadfacbcddda8ce4f?pvs=74
3 replies
38 recasts
77 reactions
Varun Srinivasan
@v
1. There is no source of truth to sync from. Messages can be added or removed from any node at any point in history due to the eventually consistent nature of CRDTs. Changes are gossiped out when they happen, but this could fail for a variety of reasons. The only way for a node to catch up 100% is to 1) sync with every other node and compare every message and 2) prevent messages from entering the network until this is completed. There are 4000 nodes x 150 million messages today with 100s of messages changing every second making this impossible.
1 reply
1 recast
19 reactions
Varun Srinivasan
@v
2. Rate limits make nodes diverge rate limits are important to protect the network since we do not charge transaction fees. global rate limiting is impossible with crdts, so they are implemented per node. It is possible for a message to be temporarily rejected from a hub due to rate limits, but accepted by others.
1 reply
0 recast
5 reactions
Varun Srinivasan
@v
3. Pruning complicates things pruning means that when one message is received another, older message might be removed. this means that older state is constantly being modified by newer messages so its hard to be efficient about comparing message ids and hard to reason about why two nodes diverge.
3 replies
0 recast
7 reactions
Varun Srinivasan
@v
4. Unidirectional sync is slow. A node can be “ahead” of another node for some accounts state and “behind” it for another account. In order for these nodes to get into sync, both of them must pull data from the other node (bidirectional sync) before any state change happens. In practice, this is challenging to implementing and we rely on unidirectional sync which means that only some state converges.
1 reply
0 recast
5 reactions
Vladyslav Dalechyn
@dalechyn.eth
Pruning really complicates things. I'm not proficient at designing systems like that but: If replaying blocks in Snapchain is the problem, given that pruning occurs way less often than adding deltas, can pruning the message be made with an additional externally generated nonce to confront at least first X bytes of the block trie to not have all blocks to be replayed, or would that involve decreasing the overall security? I feel like this can be improved.
0 reply
0 recast
0 reaction
johnjjung.eth 🛟
@jj
That’s the trade off Lucene segments are the opposite of what you’re describing, it reduces the complexity by not allowing alters so that it can be more efficient. Lucene soft deletes and has immutablity allows for concurrency and performance. Since segments don’t change it optimizes the read performance. Then it runs segment merges in the background overtime and keeps things compact and efficient. @sanjay cc
0 reply
0 recast
2 reactions
