Content
@
0 reply
0 recast
0 reaction
Ryan J. Shaw
@rjs
Update: I have doubts about shuttle I trimmed the shuttle example app down to bare bones "just write the messages" into pgsql. I'm running on a Hetzner CX52 VPS (16 cores / 32 GB RAM). The docs are vague, but imply you can run more than 1 worker. I started with 1 and observed high CPU utilization so left it there. There's no guidance on when to increase workers. There is no throughput measure I can see? In 12 hours, I have 8.7M messages in postgres. Hubble dashboard shows 502M messages, and I assume that's what I'll end up with in PG, therefore at this rate the process will complete in 29 days. I've started up 3 more workers.
3 replies
0 recast
9 reactions
Stephan
@stephancill
This was what I was seeing as well but people claim for it to be working very well, which is puzzling Gave up trying to get shuttle working and ended up building this to only index a slice of the network that I care about and has the same schema as replicator https://github.com/stephancill/lazy-indexer
1 reply
0 recast
3 reactions
Ryan J. Shaw
@rjs
Thanks for this, I'll have a look. I was starting to feel very inadequate for being unable to get reasonable performance. 16x workers and I'm still looking at 12 days to fill the DB @ 460 messages/sec. I regularly work with pushing TB of data into RDBMs, I reckon with a more batch-oriented design this could be 10-20x easily (seems to be processing 1 message per tx).
1 reply
0 recast
1 reaction
Stephan
@stephancill
I also implemented a batch oriented design for shuttle and it did speed it up significantly if you want to try it https://github.com/stephancill/farcaster-shuttle
2 replies
0 recast
2 reactions
