devops

DevOps debate of the day: where should monitoring/ alerting live in IaC? Should it live close to the component being monitored (for example, defining a DataDog alert in the same folder as the AWS service it’s monitoring) or should monitoring definitions be placed somewhere separate so it can be managed together?

cloud engineer by day, blockchain dev by night || Developer DAO member

Depends on the internal team structure and engineering culture IMO. If lots of teams managing their own services, I’d say placing all IaC with that same service is going to mean less stepping on other teams toes in my experience.

I definitely expected an”it depends” response to this 😄 but that’s what I was thinking as well, there’s certain shared monitoring that I might keep separate but want to integrate service-specific checks into their existing modules.

What worked well for our team at previous job was the monitoring team had all the infra managed and exposed namespaced things to us so e.g. we managed our own alarms and graphs in IaC in our repo, and at build/deploy time it deployed to the infra they managed. Good compromise, and if we needed more levers we collab’d

That sounds like an awesome setup. Thanks for sharing so much detail, it helps a ton!