Content pfp
Content
@
0 reply
0 recast
2 reactions

Varun Srinivasan pfp
Varun Srinivasan
@v
Anyone here worked on an eng team that processed more than 100M background jobs /day? Starting to run into issues with our setup. Looking for ideas on what people typically turn to at this scale.
28 replies
20 recasts
142 reactions

Gabriel Ayuso pfp
Gabriel Ayuso
@gabrielayuso.eth
Maybe a combo between pubsub / stream processing and batch jobs. A quick search brings me to Kafka, Flink as options to explore. Apache Beam being a good way to model the workflows decouples from the underlying engine.
1 reply
0 recast
2 reactions

Gabriel Ayuso pfp
Gabriel Ayuso
@gabrielayuso.eth
I should've left Google sooner lol. I know all the internal stuff quite well, gotta figure out the external stuff.
1 reply
0 recast
3 reactions