Content
@
0 reply
0 recast
2 reactions
Varun Srinivasan
@v
Anyone here worked on an eng team that processed more than 100M background jobs /day? Starting to run into issues with our setup. Looking for ideas on what people typically turn to at this scale.
28 replies
20 recasts
144 reactions
Gabriel Ayuso
@gabrielayuso.eth
Maybe a combo between pubsub / stream processing and batch jobs. A quick search brings me to Kafka, Flink as options to explore. Apache Beam being a good way to model the workflows decouples from the underlying engine.
1 reply
0 recast
2 reactions
Gabriel Ayuso
@gabrielayuso.eth
I should've left Google sooner lol. I know all the internal stuff quite well, gotta figure out the external stuff.
1 reply
0 recast
3 reactions
