devops

DevOps debate of the day: where should monitoring/ alerting live in IaC? Should it live close to the component being monitored (for example, defining a DataDog alert in the same folder as the AWS service it’s monitoring) or should monitoring definitions be placed somewhere separate so it can be managed together?

cloud engineer by day, blockchain dev by night || Developer DAO member

I definitely expected an”it depends” response to this 😄 but that’s what I was thinking as well, there’s certain shared monitoring that I might keep separate but want to integrate service-specific checks into their existing modules.

Depends on the internal team structure and engineering culture IMO. If lots of teams managing their own services, I’d say placing all IaC with that same service is going to mean less stepping on other teams toes in my experience.

My preferred way is have everything under one component

E.g. helm chart for Service brings its monitoring Prometheus/Alerting rules + Grafana Dashboard with it

Means I can have generic Kube Prom Stack Helm Chart and bringing in a service brings its monitoring too.
Not bringing service means all clean still

Closer to the component. Kind of similar to grouping other related resources to the component.