Farcaster Devs

Update: shuttle does not appear to be a 1:1 replacement for replicator. Docs could be much clearer on the architecture here: it seems like shuttle owns one table in pgsql, "messages", and your app can handle messages in the same tx to populate additional tables e.g. the example app populates a casts table. Not sure if I like that - would rather split the loading & transforming processes.

Update: I have doubts about shuttle

I trimmed the shuttle example app down to bare bones "just write the messages" into pgsql.

I'm running on a Hetzner CX52 VPS (16 cores / 32 GB RAM).

The docs are vague, but imply you can run more than 1 worker. I started with 1 and observed high CPU utilization so left it there. There's no guidance on when to increase workers. There is no throughput measure I can see?

In 12 hours, I have 8.7M messages in postgres. Hubble dashboard shows 502M messages, and I assume that's what I'll end up with in PG, therefore at this rate the process will complete in 29 days.

I've started up 3 more workers.

Husband, dad, dev • @dune.eth wizard • Ultimate Ⓜ️oxie Dashboard https://tinyurl.com/moxiedash • Ultimate Degen Tips FAQ https://tinyurl.com/degenfaq

it’s pretty shocking that we don’t have a better solution than this…a ~130 GB sql dump would take far less time to restore from than this. speaking of…why aren’t those available to bootstrap from?

working on infra to displace adtech at /microsub • dance music lover + host of /campfire 🎧

🎯 

Merkle should have a free solution for this if their goals are as they state..

Right this is actually what I wanted to do with my indexer

I want to start by producing a clean event stream that anybody can download from BitTorrent or S3 (if they pay me). You should be able to bootstrap in 140GB/your_download_bw seconds.