Observability is the ability to answer new questions about a running system without shipping new code.
Modern observability distinguishes itself from traditional monitoring by scope: monitoring answers pre-defined questions about known failure modes (is the CPU above 80%?), while observability lets you interrogate the system about unknown failure modes after the fact (why did checkout latency spike between 14:03 and 14:07 for users in São Paulo on Android?). The three primary data types — logs (discrete events with context), metrics (numeric aggregates over time), traces (causal chains of operations across services) — each answer different questions, and using the wrong type for the wrong question wastes both money and engineering time.
Logs: high-cardinality, high-context, expensive to store, essential for post-hoc investigation. Use for events where you need the full context later (auth failures, payment errors, unexpected code paths). Metrics: low-cardinality, aggregated, cheap to store, essential for alerting and dashboards. Use for anything you'll graph over time (RPS, latency percentiles, error rate). Traces: capture the causal chain of a single request across services, essential for debugging latency and cross-service failures. Use for anything that touches more than one service. Common mistake: logging everything at high cardinality (turning logs into low-quality metrics) or using metrics for high-cardinality data (unique_user_id as a label — will bankrupt you).
OpenTelemetry (OTel) is the CNCF standard for instrumenting code and shipping telemetry. Instrument your code once against the OTel SDK; ship the data to whatever backend you want (Datadog, Honeycomb, Grafana, self-hosted). This decouples your code from your vendor — a critical hedge as observability spend grows. Adopt OTel for new services from day one; migrate existing services opportunistically. Auto-instrumentation covers 80% of common libraries (HTTP servers, databases, queues); the last 20% requires manual spans for domain-specific operations.
Observability bills at scale look like a second infrastructure bill. Levers: (1) Sampling — head-based sampling drops X% of traces at ingest; tail-based samples based on trace properties (all errors, all >1s traces, N% of normal traces). Tail-based is more useful but requires more sophisticated collectors. (2) Log levels — INFO in production is often too verbose; move debug detail to DEBUG level and enable per-service. (3) Retention tiers — hot for 7-14 days (searchable, expensive), warm for 30-90 days (cheaper, slower search), cold for compliance-only. (4) Cardinality management — audit metric labels quarterly; drop ones that don't drive alerts or dashboards. Set a budget per service and alert engineering teams when they blow it — cost accountability is the only mechanism that works.
Vendor-managed all-in-one (Datadog, New Relic): fastest to ship, most expensive at scale. Best for small teams and mid-market. Best-of-breed (Honeycomb for tracing, Grafana + Prometheus for metrics, Loki or an ELK stack for logs): more integration work, better tools for specific jobs, cheaper at scale. OSS self-hosted (Grafana LGTM stack, SigNoz, Uptrace): lowest per-GB cost, highest operational load, requires a platform team. Reasonable progression: managed at small scale, migrate high-volume pillars (usually logs first) to self-hosted or cheaper vendors as bills grow past $100K/year.
A stack with no adoption is a bill. Practices that drive usage: (1) service owners define SLOs and dashboards during service creation, not after. (2) Every incident postmortem includes 'what observability was missing that would have caught this earlier' as a required section. (3) On-call training uses real historical incidents replayed through the observability tools. (4) A weekly or bi-weekly 'observability office hours' where the platform team helps service owners with instrumentation and queries. (5) Alerts are owned by teams, not by a central ops group — the team whose code paged them must fix, tune, or delete the alert.
Investor directory · Fundraising library · Articles A–Z · Company funding database