Farcaster Devs

Update: shuttle does not appear to be a 1:1 replacement for replicator. Docs could be much clearer on the architecture here: it seems like shuttle owns one table in pgsql, "messages", and your app can handle messages in the same tx to populate additional tables e.g. the example app populates a casts table. Not sure if I like that - would rather split the loading & transforming processes.

Update: I have doubts about shuttle

I trimmed the shuttle example app down to bare bones "just write the messages" into pgsql.

I'm running on a Hetzner CX52 VPS (16 cores / 32 GB RAM).

The docs are vague, but imply you can run more than 1 worker. I started with 1 and observed high CPU utilization so left it there. There's no guidance on when to increase workers. There is no throughput measure I can see?

In 12 hours, I have 8.7M messages in postgres. Hubble dashboard shows 502M messages, and I assume that's what I'll end up with in PG, therefore at this rate the process will complete in 29 days.

I've started up 3 more workers.

This was what I was seeing as well but people claim for it to be working very well, which is puzzling

Gave up trying to get shuttle working and ended up building this to only index a slice of the network that I care about and has the same schema as replicator

Husband, dad, dev • @dune.eth wizard • Ultimate Ⓜ️oxie Dashboard https://tinyurl.com/moxiedash • Ultimate Degen Tips FAQ https://tinyurl.com/degenfaq

Thanks for this, I'll have a look. I was starting to feel very inadequate for being unable to get reasonable performance. 16x workers and I'm still looking at 12 days to fill the DB @ 460 messages/sec. I regularly work with pushing TB of data into RDBMs, I reckon with a more batch-oriented design this could be 10-20x easily (seems to be processing 1 message per tx).

This was what I was seeing as well but people claim for it to be working very well, which is puzzling

Gave up trying to get shuttle working and ended up building this to only index a slice of the network that I care about and has the same schema as replicator 

https://github.com/stephancill/lazy-indexer

working on infra to displace adtech at /microsub • dance music lover + host of /campfire 🎧

Right this is actually what I wanted to do with my indexer

I want to start by producing a clean event stream that anybody can download from BitTorrent or S3 (if they pay me). You should be able to bootstrap in 140GB/your_download_bw seconds.

it’s pretty shocking that we don’t have a better solution than this…a ~130 GB sql dump would take far less time to restore from than this. speaking of…why aren’t those available to bootstrap from?