The problem
Operational telemetry is usually treated as a passive record of system behavior. During an incident, however, logs may arrive late, traces may be sampled, clocks may differ, and recovery actions may create gaps. The evidence used to explain an incident can itself be unreliable.
Why it was hard
Infrastructure health and evidence quality are related but not identical. A healthy collector does not prove that all relevant events were observed. A complete-looking timeline can hide missing intervals or reordered events.
How we approached it
The design question became: what evidence is required to support a conclusion, and how can the system report when that evidence is weak? This led to patterns around completeness, ordering, provenance, correlation, and confidence during incident analysis.
Tradeoffs
Collecting everything increases cost and noise. Aggressive sampling reduces both, but can remove the event that explains the failure. Evidence-aware observability makes that tradeoff explicit rather than pretending every dashboard is equally reliable.
What stayed with me
Observability should not only answer “what happened?” It should also answer “how certain are we that the available evidence is sufficient?”