A place for developers to talk about building on Farcaster.

/dev

Anyone here worked on an eng team that processed more than 100M background jobs  /day? 

Starting to run into issues with our setup. Looking for ideas on what people typically turn to at this scale.

Technowatermelon. Elder Millenial. Building Farcaster. 

nf.td/varun

Consider shifting focus from batch jobs to streaming data. You could implement websockets or leverage AWS Kinesis. Identify the primary purposes and time intervals of the jobs. Short-interval tasks should migrate to streaming data for optimal performance.

You could have had this man all to yourself but now you can’t afford him. If only you didn’t only look at staff level engineers. Maybe a lesson for future hiring. Maybe you don’t need to look further than this thread. Call me maybe when you’re hiring idea guys.