Observability retrofit

A three-month engagement adding tracing and useful metrics to a TypeScript service that had grown past the point where grepping logs worked. Nothing was rewritten. The goal was to be able to answer "why was this request slow" without shipping a deploy first.

Approach

Instrumentation went in from the edges inwards: HTTP handlers first, then the database layer, then the two external calls that turned out to account for most of the tail latency. Spans carry the tenant id and the route template, never the raw path, which keeps cardinality survivable.

• OpenTelemetry SDK, with a collector running as a sidecar
• Head sampling, plus an override header so support staff can reproduce a report at full detail
• Four dashboards, each chosen to answer a question somebody actually asks
• Alerts on symptoms rather than causes: latency and error rate at the edge

The awkward part was the existing logging. Roughly two hundred call sites wrote free-form strings, and rather than rewrite them all I added a shim that attached the current trace id to whatever was already being written. That made the old logs joinable with the new traces without a large diff.

Median latency did not move. The p99 fell by about a third once the retry loop around one of those external calls was given a budget, which is the sort of thing the traces made obvious in an afternoon and the logs had hidden for a year.