Most A/B tests at startups are underpowered, poorly analyzed, and either called too early or ignored when the result is inconvenient.
An experimentation platform assigns users randomly to variants, tracks their behavior, and computes whether observed differences are statistically meaningful. It is the closest thing product development has to a controlled trial. Done well, experiments settle debates that would otherwise drag on for months. Done badly, they produce false positives that get shipped and quietly regress revenue.
Test when: (1) the change is reversible and the wrong choice is costly; (2) you have enough traffic to detect the expected effect size in a reasonable time; (3) reasonable people at your company disagree about the outcome. Skip when: (1) the change is obviously an improvement (fixing a broken flow, meeting a compliance requirement); (2) traffic is too low to detect anything but massive effects — you'll spend 6 months testing to learn nothing; (3) the change affects only a specific customer segment already asking for it; (4) it is a strategic bet the founder wants to make regardless of first-month data. Experimentation is a tool, not an ideology.
Before starting an experiment, compute the minimum detectable effect (MDE) at your available sample size. Rough rule: to detect a 5% relative lift on a 20% baseline conversion rate at 80% power, you need ~15,000 users per variant. If your product has 1,000 signups a week, that is 15+ weeks — probably too long. Either accept detecting only large effects (>15% lift), extend the test, or don't run it. Underpowered tests are worse than no test: they produce noisy results that get overinterpreted.
Every experiment has a primary metric (the one you optimized for) and guardrail metrics (things that must not regress). Standard guardrails: signup-to-activation, weekly retention, error rate, page load latency, revenue per user. If the primary metric wins by 3% but a guardrail regresses by 5%, do not ship. Guardrails prevent the 'we A/B tested our way into a worse product' pattern where each local optimization degrades an unmeasured global metric.
(1) Peeking — checking daily and stopping when significant. This inflates false-positive rate dramatically; use sequential testing (mSPRT, always-valid p-values) if you need early stopping. (2) SRM (Sample Ratio Mismatch) — variants have unequal user counts; means the assignment or logging is broken and results are invalid. Always check. (3) Novelty effect — the new variant wins in week one because users click anything new, then reverts. Run at least one full business cycle (usually 2 weeks). (4) Multiple comparisons — testing 20 metrics; one will be significant by chance. Correct or pre-register the primary.
Publish every experiment result — wins, losses, and inconclusive — in a searchable repo. Include the hypothesis, the design, the pre-registered primary metric, and the final numbers. Inconclusive is not a failure; it is information about effect size. The team that ships winners and quietly kills losers without publishing gets a false-positive bias in institutional memory. Kill losing tests fast; extend inconclusive tests only if the potential upside justifies the traffic cost; celebrate learnings, not just wins.
Investor directory · Fundraising library · Articles A–Z · Company funding database