Muhammed Senussi
Muhammed Senussi
  • Muhammed Senussi
Three Pillars of Observability Are Not EnoughDevOps

The three pillars framing has been useful and has also let a lot of teams believe they are done once all three are installed.

The missing question

During an incident you are not really asking what the p99 was. You are asking whether checkout worked for the customer who called, and if not, where their request stopped. Answering that needs high-cardinality attributes on your spans — customer id, plan, feature flags, region — which is precisely what most metrics pipelines are designed to prevent you from storing.

If you cannot ask a question you did not anticipate, you have monitoring rather than observability.

Practical steps

  • Put a trace id in every log line and in the response headers, so support can paste one into a ticket.
  • Attach the business context to spans, not just the technical context.
  • Instrument the paths you would be asked about in a status page update, not the ones that were easy to wrap.

2 Comments

  • Daan Visser

    August 27, 2024

    Trace id in the response header is such a small thing and support love it. We added it last month.

    • Muhammed Senussi

      Muhammed Senussi

      AuthorAugust 27, 2024

      It also quietly improves bug reports. People paste the header instead of describing what they think happened.

Leave a comment