For a single service, logs are fine. You grep, you find the line, you fix it.
The moment a request touches an API, a queue, a worker, and a database, logs become a pile of disconnected sentences on four different machines with no way to know which ones belong to the same story. That's where I was: engineers spending hours reconstructing what happened to one slow request.
So I built an OpenTelemetry-native observability layer — SigNoz on ClickHouse, auto-instrumentation across FastAPI, Node workers, and Postgres, and RED metrics with p95/p99 alerting.
Here's what actually mattered.
A trace is one request's whole life
The core idea is worth stating plainly, because the tooling tends to obscure it: a trace follows one request across every service it touches, and each span within it is one unit of work with a duration and a parent.
That's it. But it changes the debugging question from "which logs relate to this?" to "what did this request do?" — and that second question has an answer you can read in one screen.
Getting there meant instrumenting Python FastAPI, Node.js workers, and PostgreSQL query execution, so a slow endpoint resolves down to the exact query holding it up rather than stopping at "the database was slow."
Sampling: keep every error, sample the boring stuff
First load test, unfiltered spans overwhelmed ClickHouse disk I/O. Full-fidelity tracing on every request is not something you can afford, and it's not something you need.
The policy that worked: capture 100% of errors and sample healthy 200s at around 10%. Errors are why you're here. Successful requests matter in aggregate — for latency distributions — and a tenth of them describes the distribution fine.
This is the observability equivalent of not printing every line of a book to find one typo.
Context propagation is where it breaks
Auto-instrumentation gets you most of the way inside a service. Between services, trace context travels in headers — and message brokers happily strip headers they don't recognize.
The symptom is a trace that ends at the queue and a second orphaned trace starting at the worker, which is exactly the boundary you most needed visibility across. Custom baggage propagators, injecting and extracting trace context through the broker, reconnected them.
If your traces look complete inside services and mysteriously truncate at every async boundary, this is your bug.
What went wrong
The OTel collector had unlimited memory buffering — the default, and fine right up until it isn't.
During a network partition between the collector and ClickHouse, the collector did exactly what it was configured to do: buffer everything it couldn't ship. It consumed all available RAM and took down adjacent microservices with it.
My monitoring stack caused an outage. There is no more humbling way to learn that observability infrastructure must degrade before the things it observes do.
The fix: a memory_limiter processor with hard limits, backpressure when it fills, and disk-backed queues so a partition means "telemetry is delayed" rather than "the server is gone."
Why ClickHouse
Over Elasticsearch, for one reason that showed up immediately: columnar storage suited the queries. Telemetry queries are analytical — p99 latency for this endpoint over six hours — not full-text lookups. ClickHouse used about 70% less memory and ran those aggregations several times faster.
I also standardized on pure OTLP rather than any vendor SDK. Backends can be swapped with zero application code changes, which is worth a lot when your monitoring vendor changes their pricing.
Results
- Every production error linked to a root-cause trace within seconds
- Continuous p95/p99 monitoring at under 1.5% CPU overhead
- Mean time to detection from hours to under 2 minutes
What I'd change today
eBPF-based tracing for network and kernel-level metrics, which collects connectivity data with no application instrumentation at all. The gap in any auto-instrumentation setup is always the code you didn't write — libraries, sidecars, the network itself — and eBPF sees that layer without asking permission from your codebase.