A place for developers to talk about building on Farcaster.

/dev

Anyone here worked on an eng team that processed more than 100M background jobs  /day? 

Starting to run into issues with our setup. Looking for ideas on what people typically turn to at this scale.

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

Interesting challenge. it heavily depends on the kind of jobs you need to process. Do they have a shared state? Are they fire & forget? CPU or IO intensive? 
So many cases…

Would love to follow the conversation on this topic.

shared state. 

think generating home feeds, updating reaction counts etc

Don't underestimate how much a single machine can handle with everything in-memory, at least for coordinating the tasks (e.g. single redis instance can handle upwards of 100k qps, could have several read replicas too), the workers can fan out as much as necessary from there.

Sounds like a good problem for stream processing. Dump activity as events on a Kafka queue and a Flink job, or multiple jobs for unrelated app functionality, can be scaled out to do the processing

...Kafka was invented at LinkedIn to handle social app updates after all

Have done this scale before - can go into detail on the AWS side with SQS + Fargate auto scale workers.