Chaos engineering is the discipline of deliberately injecting failure into production or production-like systems to discover weaknesses before real.
Chaos engineering is the practice of running controlled experiments that inject failure into a running system — killing servers, adding latency, corrupting responses, simulating regional outages — to verify the system degrades gracefully. Netflix pioneered it with Chaos Monkey in 2011; the underlying insight is that untested failure modes are functionally guaranteed to occur eventually, and finding them via deliberate experiment is dramatically cheaper than finding them via real incident.
The Principles of Chaos Engineering document defines the loop: (1) Define steady state — measurable properties of a healthy system (checkout success rate, p99 latency). (2) Hypothesize the steady state continues under some failure — 'if we kill one API replica, success rate stays above 99.5%.' (3) Introduce the failure in a controlled scope. (4) Compare actual vs. hypothesized. Experiments that confirm the hypothesis build confidence; experiments that disprove it uncover bugs to fix. Skip the hypothesis and you're not doing chaos engineering, you're just breaking things.
Netflix-scale automated chaos is not the right starting point for a 20-engineer team. Start with quarterly game days: a scheduled 2-hour block where the team meets, picks a failure scenario ('the primary database becomes unreachable'), and simulates it in staging or against a controlled production slice. Discuss what breaks, fix the biggest gaps, run the same scenario again next quarter to verify improvement. Game days build the muscle for real incidents and surface the same class of bugs as automated chaos, at a fraction of the setup cost.
Common experiments in order of value/difficulty: (1) Instance termination — kill a random pod/VM, verify autoscaling and load balancing recover. (2) Network latency — add 500ms to calls to a specific dependency, verify timeouts and circuit breakers work. (3) Dependency unavailability — return errors from a downstream API, verify graceful degradation. (4) Resource exhaustion — fill disk, saturate CPU, verify alerts fire and the system doesn't cascade. (5) Region/AZ failure — simulate loss of a whole availability zone, verify failover. Start with #1 and #2; only advanced teams need #5.
Chaos experiments must be scoped so a failed experiment doesn't take down customer traffic. Techniques: run in a copy of production with mirrored traffic; run against a small percentage of users with a fast rollback switch; run only during business hours with the team watching; have an explicit abort mechanism that returns the system to normal in under a minute. Chaos experiments that themselves cause outages are how the practice gets banned; strict blast-radius discipline is what makes it sustainable.
Gremlin — commercial platform, wide range of experiments, best UX. Chaos Mesh, Litmus — Kubernetes-native, open source. AWS Fault Injection Service — AWS-managed, integrates with EC2/RDS/ECS. ToxiProxy — application-layer network fault injection, good for local dev testing. Netflix's original Chaos Monkey — still available, narrower scope. For most teams, starting with game days and simple scripts against your staging environment is enough; adopt a platform when the practice justifies the investment.
Investor directory · Fundraising library · Articles A–Z · Company funding database