Observability is the property of a system where its internal state can be inferred from its external outputs — logs, metrics, and traces.
Observability is the ability to answer arbitrary questions about a system's behavior from the outside, without needing to ship new code. It's an evolution of monitoring — monitoring tells you that a known thing is broken; observability lets you investigate unknown things that are breaking. The three canonical pillars are logs (discrete events), metrics (aggregated numbers over time), and traces (the path of a request through your system). Charity Majors' writing and Google's SRE book are the reference points.
Logs are cheap to emit, expensive to query at scale, and useful for detailed forensics on individual requests. Metrics are cheap to store and query but lossy — you can see that p99 latency spiked, not why. Traces show causality across services and are indispensable once you have more than 2-3 services in a request path but expensive to store at full fidelity (hence sampling). Most companies over-invest in logs, under-invest in traces, and treat metrics as dashboards rather than alerting substrate. The right investment order for a growing team: structured logs first, then metrics, then traces once you're distributed.
A log line like 'user 12345 hit /api/checkout in 234ms' is human-readable and machine-hostile. The same event as JSON — {event: 'request', user_id: 12345, path: '/api/checkout', duration_ms: 234, trace_id: '...'} — is queryable, filterable, and joinable. Adopt structured logging early; retrofitting a codebase full of printf logging later is one of the most tedious refactors in engineering. Every log line should carry request_id/trace_id, user_id (when authenticated), and enough context to reproduce the situation without opening the code.
Traditional metrics systems (statsd, Prometheus) struggle with high-cardinality dimensions — labels like user_id or request_id that can take millions of values. But most real debugging questions are high-cardinality ('which specific customer is seeing 500s?'). Modern observability tools (Honeycomb, DataDog with distribution metrics, Grafana with Loki/Tempo) are designed for this. When evaluating tools, the cardinality ceiling is often the difference between 'we can answer the question in 30 seconds' and 'we need to ship a new log line and wait a week.'
Full-fidelity tracing at scale is prohibitively expensive. Two sampling approaches: head-based (decide at request start based on random chance — simple, but misses rare interesting traces) and tail-based (decide after the trace completes based on whether it was slow or errored — captures the interesting stuff, requires a collector layer). Tail-based sampling is worth the complexity once you're past a certain volume; head-based is fine below that. Always sample errored requests at 100%.
Observability data becomes actionable when tied to service level objectives (SLOs). 'API p99 latency should be under 400ms 99% of the time over a 30-day window' is a target you can build alerts and dashboards around. Without SLOs, teams end up alerting on every anomaly and dashboards become abstract art. See the SLO page for the framework; observability tooling exists to serve the SLO conversation, not the other way around.
Investor directory · Fundraising library · Articles A–Z · Company funding database