Farcaster

One point of feedback we got on the Snapchain design doc is that we didn't fully explain why sync in the current model is a problem. 

@sanjay wrote up a good thread on our internal slack, which i'll try to summarize below.

One point of feedback we got on the Snapchain design doc is that we didn't fully explain why sync in the current model is a problem. 

@sanjay wrote up a good thread on our internal slack, which i'll try to summarize below. 

https://warpcast.notion.site/Snapchain-v2-Public-10e6a6c0c101807aadfacbcddda8ce4f?pvs=74

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

1. There is no source of truth to sync from. 

Messages can be added or removed from any node at any point in history due to the eventually consistent nature of CRDTs. Changes are gossiped out when they happen, but this could fail for a variety of reasons. The only way for a node to catch up 100% is to 1) sync with every other node and compare every message and 2) prevent messages from entering the network until this is completed. There are 4000 nodes x 150 million messages today with 100s of messages changing every second making this impossible.

3. Pruning complicates things 

pruning means that when one message is received another, older message might be removed. this means that older state is constantly being modified by newer messages so its hard to be efficient about comparing message ids and hard to reason about why two nodes diverge.

4. Unidirectional sync is slow. 

A node can be “ahead” of another node for some accounts state and “behind” it for another account. In order for these nodes to get into sync, both of them must pull data from the other node (bidirectional sync) before any state change happens. In practice, this is challenging to implementing and we rely on unidirectional sync which means that only some state converges.

The learning here is that eventual consistency (CRDTs) in a network with a large number of write nodes and high volume of messages is never going to be good at being real-time. 

You either have to accept that some things will be delayed or out of sync for a really long time, or pick a different consistency model

2. Rate limits make nodes diverge

rate limits are important to protect the network since we do not charge transaction fees. global rate limiting is impossible with crdts, so they are implemented per node. It is possible for a message to be temporarily rejected from a hub due to rate limits, but accepted by others.