waypoint

Boys only want one thing and it's to watch every single available thread on their laptop be used for their worker/queue system written in Rust experience zero memory leaks and negligible CPU usage while it backfills from multiple Farcaster hubs.

Based House. Engineering @neynar. Amusement, wonder, and uncertainty ✴︎ libred.org

Reason for Rust was so it could share all these resources on the same cluster/node/machine.

But I am running an army of workers against the queue using 700 connections (max allowed for CloudSQL without blowing all our budget).

Will report back how long it takes. 🫡

0.1 FIDs per second 😆

Happy to hear any advice you might have. Everything is indexed well, Postgres is humming along. Bulk insertion of messages would be probably be the biggest leg up, as well? Haven't considered it due to rollbacks.

Even perfect Rust code can’t make hub backfill fast

Good shit brother - this has come so far!

I make lights blink and play with crypto (cryptography and cryptocurrency) at /neynar. they/them or he/him