A place for developers to talk about building on Farcaster.

/dev

Any real-time or near-real-time indexer with high reliability and low data latency needs three high-level components. 

1. Stream indexing - for delivering on the real-time promises. 

2. Complete backfill - ideally this functionality runs only once per sink

3. Time-Window backfill - ensures the ability to recover from downtimes on #1 by minimizing real-time guarantee violations. (might not need this if the data set is small and the sink can afford to handle repeated #2)

Seems on top of 1,2 and 3 we are also building 4: do steps 1-3 at the same time @runninyeti.eth

Monkey on the chain | wrangling data at The Indexing Company
https://www.indexing.co/

can you elaborate?

I don't fully understand the need to run 2 and 3 in parallel. 2 is a superset of 3.

I see your point and I think it is in the details. 

Although 2. the complete backfill can be a one time trigger. 3. for the time window can be automated if the system is down but knows the time period is was down. No need to automate for 3.

Then there is a case where shipping a whole history can be easier vs. 3 where filtered on specific data(contracts/tokens/etc.) for a specific period where only that data is active. For the latter you need to know when these tx happened.