Observability: Ask Production Better Questions

Monitoring asks whether known bad conditions are happening. Observability helps you investigate conditions you did not predict. The useful test is simple: when one customer says checkout hung at ten fourteen, can you find that request, see each dependency it called, and explain where the time went?

User symptom: checkout took 18 seconds

Questions:
Which requests were affected?
Did errors or latency change by region or release?
Which operation consumed the time?
Which dependency returned slowly or failed?
Was the user impact inside the reliability objective?
What changed just before the symptom?

Collect telemetry to answer operational questions, not because three signal types appear in a vendor diagram. A metric reveals a population-level spike. A trace locates slow work in representative requests. A log preserves detailed events and errors. Their shared service, environment, release, route, and trace identifiers turn separate records into evidence.

A useful log event has a stable name, timestamp, severity, service, environment, release, request or trace identifier, outcome, duration, and bounded business context. Write structured JSON so machines can filter fields without parsing sentences. Keep the human message, but do not make it the schema.

{
  "timestamp": "2026-07-25T10:14:32.481Z",
  "level": "error",
  "event": "payment_authorization_failed",
  "service": "checkout",
  "release": "2026.07.25.3",
  "trace_id": "4f92c6...",
  "order_id": "ord_8132",
  "provider": "acquirer_a",
  "duration_ms": 3002,
  "error_type": "deadline_exceeded"
}

Log once at the boundary that can add useful context. If four layers catch and rethrow the same exception, four identical stack traces raise cost without adding evidence. Preserve the original cause when wrapping errors. Do not log passwords, session cookies, authorization headers, access tokens, full payment data, or unreviewed request bodies.

Counters only increase and suit requests, failures, and jobs. Gauges move up and down and suit queue depth or memory in use. Histograms count observations in buckets and let you aggregate latency across instances. Use rates for counters. A raw cumulative request count mostly tells you how long the process has been alive.

http_requests_total{service="checkout",route="/orders/{id}",status_class="5xx"}
http_request_duration_seconds_bucket{service="checkout",route="/orders/{id}",le="0.5"}
queue_depth{service="fulfillment",queue="shipments"}
build_info{service="checkout",release="2026.07.25.3"} 1

# Useful views
error rate = 5xx request rate / all request rate
p95 latency from histogram buckets
queue age alongside queue depth

Averages hide the users waiting in the tail. Report a latency distribution and inspect the ninety-fifth or ninety-ninth percentile when the traffic volume supports it. Choose histogram boundaries around product thresholds, not round numbers inherited from another service. A thirty second job and a fifty millisecond API need different buckets.

A trace represents one end-to-end operation. Spans represent timed pieces of work, with parent-child relationships, attributes, events, and status. The incoming server span should contain child spans for database, cache, queue, and outbound HTTP calls. The shape shows where time accumulated and where an error first appeared.

POST /checkout                         3.24 s
├─ validate basket                     0.02 s
├─ SELECT inventory                    0.08 s
├─ POST payment-provider /authorize    3.01 s  ERROR
└─ publish order-failed                0.03 s

trace_id: shared by every span
span_id: unique operation
parent_span_id: causal relationship
baggage: propagated context; use sparingly

Context propagation is the part teams underestimate. Forward the standard trace context through HTTP and messaging libraries. For async work, link the consumer span to the producer context. Do not invent a trace ID header if your instrumentation already supports a standard one. Never place secrets or unbounded payloads in span attributes.

Start where work enters and leaves the service. Instrument HTTP servers and clients, database queries, cache calls, message producers and consumers, scheduled jobs, and external providers. These boundaries reveal latency, errors, retries, and fan-out. Adding a span around every helper function creates noise and overhead without explaining the system.

Minimum service telemetry:
request count, outcome, and duration
in-flight work and queue age
dependency count, outcome, and duration
retries and timeouts
resource saturation
release and configuration version

Useful span attributes:
service.name, deployment.environment, service.version
http.request.method, http.route, http.response.status_code
db.system, db.operation.name
messaging.system, messaging.destination.name

Prefer automatic instrumentation for standard libraries, then add manual spans around business operations such as price calculation or payment authorization. Review generated route names and database statements before shipping. Parameterized route templates are safe dimensions; raw URLs and SQL values can expose data and explode cardinality.

A service level indicator measures user-visible success, such as the proportion of checkout requests that complete correctly within two seconds. A service level objective sets the target over a window. The error budget is the allowed unsuccessful fraction. It supports an explicit trade: spend reliability on change, but slow down when failures consume the budget too quickly.

SLI = good checkout requests / eligible checkout requests
SLO = 99.9% over a rolling 28-day window

Allowed bad fraction = 0.1%
At 10,000,000 eligible requests:
error budget = 10,000 bad requests

Burn rate = observed bad rate / allowed bad rate
burn rate 1  → budget consumed exactly across the window
burn rate 14 → budget disappearing fourteen times too fast

Define eligible and good precisely. Exclude health checks and requests rejected before entering the product flow. Decide how cancellations and provider failures count from the user's perspective, not from team ownership. A dependency-caused failed checkout is still a failed checkout.

A page should mean a person must act now. Alert on fast error-budget burn, sustained user-visible latency, or a backlog that will soon miss its deadline. High CPU, one crashed replica, or low disk may matter, but they are usually cause signals. Page on them only when immediate action prevents known impact.

PAGE:
Checkout SLO burn rate is 18 over 5 minutes
and 7 over 1 hour, affecting 3 regions.
Runbook: /runbooks/checkout-slo
Dashboard: /d/checkout
Recent releases: 2026.07.25.3 at 10:08 UTC

TICKET:
Search cluster disk projected to reach 80% in 4 days.
Owner: search-platform
Capacity procedure: /runbooks/search-capacity

Use a short and a long window together. The short window catches a severe outage quickly; the long window prevents paging on a one-minute blip. Every alert needs an owner, impact statement, current evidence, and first diagnostic action. If the receiver cannot act, route it elsewhere or remove it.

Each unique metric label combination creates a time series. A label with user IDs, request IDs, raw URLs, stack traces, or timestamps can create millions of series and overload storage or query systems. Metrics need bounded dimensions. Put high-cardinality identifiers in sampled traces or indexed logs where you can query them deliberately.

# Bad: one series per user and raw path
http_requests_total{user_id="813294",path="/orders/998123"}

# Better: bounded route and status class
http_requests_total{route="/orders/{id}",status_class="2xx"}

# Estimate series before shipping
services × routes × methods × status_classes × regions
40 × 80 × 5 × 5 × 4 = 320,000 possible series

Control cost with retention tiers, sampling, aggregation, and ownership. Keep detailed telemetry where it answers active questions. Downsample older metrics. Retain security and audit records according to their separate policy. “Store everything forever” is not an observability strategy.

Suppose checkout latency jumps after a release. Start with impact, not random host inspection. Confirm affected routes, regions, releases, and time range. Compare successful and failed traces. In the slow traces, the payment span runs for three seconds and ends with a deadline error. Now inspect dependency outcomes and retry count.

rpc error: code = DeadlineExceeded desc = context deadline exceeded

10:08  release 2026.07.25.3 begins
10:11  checkout p95: 0.8 s → 6.4 s
10:11  payment attempts/request: 1.0 → 2.8
10:11  payment deadline errors increase

Trace evidence:
attempt 1: 3.0 s deadline
attempt 2: 3.0 s deadline
request deadline: 6.5 s

The raw text tells you the local operation exceeded its context deadline. It does not prove the provider was down. The trace proves two sequential waits. The release marker and rollback prove the regression follows the new retry behavior. Good incident work separates observation, hypothesis, test, and conclusion.

Build from the user journey inward. One carefully instrumented checkout path is worth more than host dashboards across fifty services when nobody can connect those hosts to a failed order.

BUILD
1. Name critical user journeys and owners
2. Define good events and an SLO for each journey
3. Measure rate, errors, duration, and saturation
4. Instrument inbound, outbound, database, cache, and queue boundaries
5. Propagate trace context across HTTP and messaging
6. Add structured events with safe identifiers and release version
7. Link metrics to traces and traces to logs
8. Set retention, sampling, redaction, and series budgets
9. Test telemetry loss; the product must keep working
10. Page on fast user-impact burn with a usable runbook

INCIDENT
1. State user impact and exact time window
2. Check SLI, traffic, regions, routes, and release markers
3. Compare healthy and unhealthy populations
4. Open representative traces; locate the slow or failed span
5. Query correlated logs by trace ID and event
6. Check dependency, retry, queue, and saturation metrics
7. Mitigate with rollback, flag, traffic shift, or capacity
8. Verify user metrics recover
9. Preserve timestamps, queries, traces, and decisions
10. Fix the missing test and missing signal

The useful standard is not perfect telemetry. It is a short path from a user symptom to a defensible explanation. Keep identifiers connected, dimensions bounded, alerts actionable, and release changes visible. Then production can answer questions instead of merely producing charts.