A place for developers to talk about building on Farcaster.

/dev

Any real-time or near-real-time indexer with high reliability and low data latency needs three high-level components. 

1. Stream indexing - for delivering on the real-time promises. 

2. Complete backfill - ideally this functionality runs only once per sink

3. Time-Window backfill - ensures the ability to recover from downtimes on #1 by minimizing real-time guarantee violations. (might not need this if the data set is small and the sink can afford to handle repeated #2)

How do you ensure data consistency and avoid duplication when integrating complete backfill with real-time stream indexing, particularly during time-window backfill recovery?

uniqueness constraints, foreign keys, primary keys