Content
@
0 reply
0 recast
0 reaction
Classicus
@classicus
Gm Anyone ever create an executable instance to run dbt pipelines in the cloud? If so, curious to know how you handled concurrent/parallel requests from your orchestration tool (e.g Airflow, CRONs, โฆ)
1 reply
0 recast
0 reaction
๐๐๐๐๐ฅท๐ฝ
@jazzlisp.eth
GM. What do you mean by concurrency? Like on what level, DBT job level, DAG level, or task level? Maybe more specific on the requirement or the actual problem?
1 reply
0 recast
0 reaction
Classicus
@classicus
Concurrency at the task level: two DAGs calling a single instance at once, hence running two jobs with different parameters with the same underlying resources. Weโve encountered hiccups unless we make it 1:1, curious about other infra layouts. (Instance in our case is Cloud Run container, but trying to generalize)
0 reply
0 recast
0 reaction
