Backend

Gm

Anyone ever create an executable instance to run dbt pipelines in the cloud?

If so, curious to know how you handled concurrent/parallel requests from your orchestration tool (e.g Airflow, CRONs, …)

Backend-Crypto Nerd 🤓
Running to stay sane🏃‍♂️
Video games to stay young 🎮

🎞️ Art Photography 🎥 Filmmaking
🧑🏽‍💻 MPP/Distributed System/Data Analytics
🎷 Jazz 🎹 HipHop
🚀 Failed 1 company, sold 1, and building 1

Concurrency at the task level: two DAGs calling a single instance at once, hence running two jobs with different parameters with the same underlying resources.

We’ve encountered hiccups unless we make it 1:1, curious about other infra layouts.

(Instance in our case is Cloud Run container, but trying to generalize)

GM.
What do you mean by concurrency? Like on what level, DBT job level, DAG level, or task level? 
Maybe more specific on the requirement or the actual problem?