Content
@
0 reply
0 recast
0 reaction
Varun Srinivasan
@v
Anyone here worked on an eng team that processed more than 100M background jobs /day? Starting to run into issues with our setup. Looking for ideas on what people typically turn to at this scale.
25 replies
25 recasts
160 reactions
Ethspresso ππ΅πΉπ©
@ethspresso.eth
Airflow + K8s on bare metal is serving us well. At smaller scale than this though, something like 100k+ daily jobs of varying size and runtime. We use this for everything from ML training jobs to data transformations and 3rd party integrations. As @jacek mentioned, streaming is a good idea, for certain use-cases.
1 reply
0 recast
2 reactions
Samuel
@samuellhuber
regarding Airflow in K8s at Scale here's Shopify doing it https://shopify.engineering/lessons-learned-apache-airflow-scale
0 reply
0 recast
1 reaction
