A place for developers to talk about building on Farcaster.

/dev

Any real-time or near-real-time indexer with high reliability and low data latency needs three high-level components. 

1. Stream indexing - for delivering on the real-time promises. 

2. Complete backfill - ideally this functionality runs only once per sink

3. Time-Window backfill - ensures the ability to recover from downtimes on #1 by minimizing real-time guarantee violations. (might not need this if the data set is small and the sink can afford to handle repeated #2)

Great points. Do you think there’s ever a case where time window backfill might introduce more complexity than it’s worth!

If you need the reliability, you take on the complexity

That makes sense, reliability often demands complexity. I wonder if there are any patterns or best practices to keep the time window backfill manageable