Content
@
0 reply
0 recast
2 reactions
Varun Srinivasan
@v
Anyone here worked on an eng team that processed more than 100M background jobs /day? Starting to run into issues with our setup. Looking for ideas on what people typically turn to at this scale.
28 replies
20 recasts
143 reactions
Tiago
@alvesjtiago.eth
depends on the requirements for the background jobs but currently running more than that with a setup of Kafka and Golang stream processors for fast processing ones
1 reply
0 recast
1 reaction
Tiago
@alvesjtiago.eth
also using temporal.io self hosted for longer running critical ones (harder to manage at that scale because the best performing setup includes Cassandra)
1 reply
0 recast
1 reaction
