Muhammed Senussi
Muhammed Senussi
  • Muhammed Senussi
Infrastructure as Code Only Works If Nobody ClicksDevOps

The value of declarative infrastructure comes entirely from the declaration being true. Drift does not degrade that value gradually; it destroys it, because a repository you cannot trust is worse than no repository at all.

Accept that the click will happen

Somebody will change something by hand at three in the morning, and they will be right to. The failure is not the click, it is the absence of a path back. Build one:

  • Run drift detection on a schedule and alert on it like any other production signal.
  • Make the console read-only for humans in normal operation, with a documented break-glass role that is loud when used.
  • Treat every break-glass session as an incident action item: reconcile within the week or the exception becomes the system.

Plan output belongs in the pull request. A reviewer approving Terraform without seeing the plan is approving a diff in a language they cannot execute.

3 Comments

  • Sanne de Vries

    May 28, 2024

    Scheduled drift detection changed our culture more than any policy did. Being visible was enough.

  • Yara Mansour

    May 30, 2024

    How loud is "loud" for break-glass? We are worried about alert fatigue.

    • Muhammed Senussi

      Muhammed Senussi

      AuthorMay 30, 2024

      A message in the team channel naming the person and the role, nothing paging. The point is visibility, not punishment. It happens maybe twice a quarter, so there is no fatigue to speak of, and knowing it will be seen is enough to stop the casual use.

Leave a comment