this thing (Warpcast's hub replicator) is really good, kinda curious how the merge happens when the data comes in

And it's done per batch (the yellow part)?

So roughly:
1. pull data, store it in some kind of "pool"
2. every X minutes, merge with DB
3. GOTO 1

Building @warpcast and @farcaster. Previously Coinbase, Brigade, Causes.

We’re using SQL’s “ON CONFLICT” clause specify how to merge data based on some key. If new data arrives that conflicts, it will either be ignored (if for some reason it’s an old message, for example) or we’ll update the existing data.