Why Does Your A/B Test Keep Lying to You?
Teams ship a new onboarding, watch the dashboard for two days, see a green number, and call it a win. Then the win evaporates by week three and nobody can explain why. The experiment didn't fail — it was never real. Here is how a test quietly lies, and what actually makes a result trustworthy.
Q: Why is "I see a lift, so I'll ship it" the most expensive habit in mobile?
It's the peeking problem. If you check a test daily and stop the moment it crosses significance, your real false-positive rate climbs from the nominal 5% to well over 40%. You're not testing a hypothesis, you're hunting for the day noise looked like signal. Then there's multiple comparisons: you glanced at ten metrics and one lit up — that's expected from chance alone. And the novelty effect: players poke at the new thing for a week because it's new, not because it's better. A lift you "see" on day three is usually one of those three wearing a costume.
Q: How do you size a test before you run it?
With a power analysis, not a guess. You need your baseline conversion rate, the smallest effect you care about (the MDE), your significance level, and your tolerated false-negative rate — and those four spit out how many users per arm and how many days. No baseline? Don't test yet; collect data first. A casual game with 30% D1 retention trying to detect a 1-point move needs tens of thousands of users per arm, which a small team simply can't feed in a reasonable window — so either test a bigger lever or accept a long run. "I have 100k DAU" is not a sample size; what matters is how many land in each variant after split.
Q: What is a holdout, and what is a confounder?
A holdout is a slice of users who never receive the experiment, so you can measure the long-term and network effects — the spillover where changing one group quietly shifts a shared metric. A confounder is something happening at the same time: a holiday, a UA burst, a version release. Your variant "won" maybe because Christmas did. The novelty-vs-real-retention question is only answerable by extending the window and watching the holdout. Without those, you've measured correlation with a calendar you didn't account for.
Q: How does a small team run experiments without fooling itself?
Pre-register the hypothesis and the stopping rule before you launch, and don't abandon it because day two looked pretty. Use sequential testing that spends your error budget correctly instead of naive peeking. Watch guardrail metrics — crash rate, retention, the distribution of ARPU, not one happy average. And resist testing twenty tiny tweaks hoping one sticks; a big, well-formed bet beats a pile of underpowered micro-tests that each lie a little.
| Mistake | What it does | The fix |
|---|---|---|
| Peek and stop | false positives >40% | pre-registered stop rule |
| Many metrics, one "win" | chance lights one up | correct for comparisons |
| No baseline | can't size the test | gather data first |
| Ignore confounders | credit the calendar | holdout + longer window |
