A place for developers to talk about building on Farcaster.

/dev

Anyone here worked on an eng team that processed more than 100M background jobs  /day? 

Starting to run into issues with our setup. Looking for ideas on what people typically turn to at this scale.

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

Interesting challenge. it heavily depends on the kind of jobs you need to process. Do they have a shared state? Are they fire & forget? CPU or IO intensive? 
So many cases…

Would love to follow the conversation on this topic.

shared state. 

think generating home feeds, updating reaction counts etc

Sounds like a good problem for stream processing. Dump activity as events on a Kafka queue and a Flink job, or multiple jobs for unrelated app functionality, can be scaled out to do the processing

...Kafka was invented at LinkedIn to handle social app updates after all

Freelance data engineer | Backend and data engineering for Spotify, Candy Crush, helped build SF-based appsec startup | Building web3gains.com | saarw.twitter

And yes, I've worked at King when we processed events for 100M DAU with Flink.

If you're running in GCP, like Spotify, their Dataflow infra also offers similar functionality that works at that scale