A place for developers to talk about building on Farcaster.

/dev

Any real-time or near-real-time indexer with high reliability and low data latency needs three high-level components. 

1. Stream indexing - for delivering on the real-time promises. 

2. Complete backfill - ideally this functionality runs only once per sink

3. Time-Window backfill - ensures the ability to recover from downtimes on #1 by minimizing real-time guarantee violations. (might not need this if the data set is small and the sink can afford to handle repeated #2)

Monkey on the chain | wrangling data at The Indexing Company
https://www.indexing.co/

can you elaborate?

I don't fully understand the need to run 2 and 3 in parallel. 2 is a superset of 3.

Seems on top of 1,2 and 3 we are also building 4: do steps 1-3 at the same time @runninyeti.eth

uniqueness constraints, foreign keys, primary keys

How do you ensure data consistency and avoid duplication when integrating complete backfill with real-time stream indexing, particularly during time-window backfill recovery?

If you need the reliability, you take on the complexity

Great points. Do you think there’s ever a case where time window backfill might introduce more complexity than it’s worth!