Day 11 built confidence intervals. Day 12 built hypothesis tests. A/B testing is where both of those tools actually come to life: the process companies use to make real product decisions through controlled experiments instead of gut feeling.
Day 12 ended with a solid hypothesis testing framework. A/B testing is that same framework, plus a set of practical questions the theory alone never answers. How do you actually decide which users get control and which get treatment? How long do you let the experiment run? What do you do when the p-value changes every time it gets checked? What if one group ends up with more high-value users just by chance? How do you handle experiments where users can influence each other?
These are exactly the questions that separate someone who understands statistics from someone who can actually run an experiment people trust. This article covers both halves.
Five companies, five product decisions, five experiments.
Netflix is testing a continuous autoplay feature, the next episode starts automatically after a 10-second countdown instead of waiting for a manual click. The current next-episode click rate (people who voluntarily start the next episode within 5 minutes of finishing one) is 52%. Netflix's product team believes autoplay could push this to at least 57%.
The experiment: 4,000 users in control (old experience), 4,000 in treatment (autoplay on).
The question: "Is the lift in next-episode engagement real, and big enough to justify the UX change, given some users might find autoplay annoying?"
Uber's driver experience team redesigns the surge pricing notification screen, adding an estimated earnings preview and a small map of demand hotspots. The current driver acceptance rate on surge trips is 68%. They want to catch any improvement of at least 3 percentage points.
The experiment: 3,500 drivers see the old surge screen (control), 3,500 see the new design (treatment).
The question: "Does knowing their projected earnings before accepting actually make drivers more likely to take surge trips?"
Swiggy's data science team builds a new restaurant ranking algorithm that personalizes results based on a user's past orders, time of day, and current weather. The current order completion rate (users who actually place an order in a session) is 71%. This one runs as a one-tailed test, since the team only cares whether the new ranking is better, not just different.
The experiment: 5,000 users see the current ranking (control), 5,000 see the personalized ranking (treatment).
The question: "Does personalized ranking actually lead to more people completing an order?"
Spotify's growth team redesigns the new-user onboarding flow, replacing a generic "tell us what you like" genre picker with a "play three songs and we'll tune to you" interactive experience. Current 30-day retention for new users is 38%. They want to catch a meaningful change in either direction here, since the new flow could just as easily confuse people, so this one's a two-tailed test.
The experiment: 6,000 new users go through the old onboarding (control), 6,000 go through the new one (treatment).
The question: "Does an interactive onboarding experience actually keep new users around longer?"
Google's Search team redesigns the visual layout of result snippets, a bolder title font, a cleaner URL display, a subtle favicon. Current click-through rate (CTR) on the first result is 3.1%. With billions of searches a day, even 0.1 percentage points matters here.
The experiment: 80,000 search sessions see the old layout (control), 80,000 see the new card design (treatment).
The question: "Does the new card design actually increase CTR, and is that lift practically meaningful at Google's scale, or just a statistical artifact of a massive sample size?"
An A/B test is a randomized controlled experiment. Take a population, users, trips, orders, sessions, split it randomly into two groups, show each group a different experience, and measure whether the difference in outcomes is real or just noise.
"A" is the control, the current experience, the status quo. "B" is the treatment, the change actually being tested.
The random part is the whole ballgame. Without it, a difference in outcomes can't actually be attributed to the change, it could just as easily be because the two groups happened to be different kinds of people to begin with.
Think of it like testing whether a new fertilizer makes plants grow taller. Put the new fertilizer on plants in the sunny corner of a greenhouse and the old one on plants in the shade, and any height difference could be the fertilizer, or it could just be the sunlight, the variables have gotten mixed up. The fix: randomly assign fertilizer A or B to each plant regardless of where it happens to sit. Now any consistent height difference has to come from the fertilizer. That's exactly why randomization sits at the heart of A/B testing.
Before companies adopt real A/B testing, a common approach is comparing a metric before launching a feature to the same metric after. It sounds reasonable. It's actually deeply flawed.
Netflix's watch time in January versus February will differ because of new content releases, seasonal patterns, and app updates, not just whatever algorithm change happened to ship in between. The effect of a change can never be cleanly separated from everything else that changed at the same time. A controlled experiment run in parallel avoids this entirely, both groups live through the exact same external world at the exact same time.
Before writing a single line of code, four structural decisions decide the quality of everything that follows.
This is what actually gets randomly assigned, and it isn't always "a user."
| Product | What gets assigned | Why |
|---|---|---|
| Netflix | User account | Watch behavior is account-level |
| Uber | Driver ID / Rider ID | Trip decisions are per person |
| Swiggy | User session or device ID | Some users have multiple devices |
| Spotify | User ID | Listening patterns are per user |
| Google Search | Search session (cookie) | Same user may want a consistent experience |
One rule matters more than the others: the randomization unit has to sit at the same level as the metric being measured. Randomize by session but measure by user, and users with lots of sessions get over-represented, which throws off the variance estimates.
Control (A): the existing experience, also called the baseline. This is the null hypothesis, wearing a product hat. Treatment (B): the new experience. This is the alternate hypothesis, wearing the same hat.
More than one treatment can run at once (A/B/C testing), but each extra variant shrinks the sample available to every group and adds a multiple testing problem on top. For most early-stage experiments, one treatment is plenty.
This is where most experiments actually fail, not in the analysis, in how the metric gets defined in the first place.
The primary metric is the one number that answers the question. Every A/B test needs exactly one, chosen before the experiment starts. Pick it after seeing the results and that's p-hacking. Guardrail metrics need to stay stable, they catch collateral damage. Secondary metrics just help explain what's happening, they don't get to decide anything.
| Company | Primary metric | Guardrail metrics | Secondary metrics |
|---|---|---|---|
| Netflix | Next-episode click rate | Unsubscribe rate, complaints | Session length, ratings |
| Uber | Driver acceptance rate | Trip cancellation rate, app crashes | Earnings per trip, ratings |
| Swiggy | Order completion rate | Session abandonment rate | Items per order, avg order value |
| Spotify | 30-day retention | Skip rate, complaint rate | Playlist saves, feature discovery |
| CTR on result 1 | Time-to-click (too fast means accidental) | Scroll depth, bounce rate |
A guardrail is what actually makes a "winner" a winner. Netflix's autoplay might raise next-episode rates while also raising unsubscribes, that's not a win, that's a trade-off, and it needs a leadership call, not a quiet ship decision.
How long the experiment runs has to be decided before it starts.
A few rules of thumb: run for at least one full week to capture weekday and weekend behavior, since people behave differently on each. Run for at least two weeks for new-user experiments, to account for different signup days. Never stop early just because a significant result happened to show up, that's the peeking problem, coming up shortly. And never extend the experiment just because significance wasn't reached either, that's p-hacking wearing a different disguise.
Here's the full, industry-standard process. Steps 1 through 5 all happen before a single data point gets collected.
Step 1: Define primary, guardrail, and secondary metrics. Step 2: State hypotheses (H₀ and H₁). Step 3: Choose α and target power. Step 4: Calculate required sample size (minimum detectable effect). Step 5: Determine experiment duration.
Step 6: Randomize and run. Do not peek.
Step 7: Check sanity, run the statistical test. Step 8: Make the decision (ship / no-ship / iterate).
This is the single most skipped step in amateur experimentation, and the single most important one in the professional version. Running an underpowered experiment is a bit like going to court with too little evidence, the guilty usually walk (a real effect goes undetected) and everyone's time gets wasted doing it.
When the primary metric is a proportion, click rate, conversion rate, retention, which is most of them in tech, the sample size formula per group looks like this:
The two terms in the numerator are each doing a different job. The first (zα/2) makes sure there's enough data to catch false positives at the rate actually chosen. The second (zβ) makes sure there's enough data to catch real effects at the power wanted. Both have to be satisfied at once, exactly why they get added together rather than picked between.
That zβ=0.84 figure isn't a rule of thumb, it comes directly from the power asked for. Power is the probability of correctly detecting a real effect, and 80% power means wanting the true effect to clear the detection threshold 80% of the time. The z-score sitting at the 80th percentile of a standard normal distribution is 0.84 (stats.norm.ppf(0.80)), so that's the value going into the formula. Ask for 90% power instead and zβ becomes 1.28, a bigger number, pushing the required sample size up too, since higher power needs more data, the same way higher confidence needs a bigger zα.
Before computing a sample size, one question needs an answer: what's the smallest improvement actually worth caring about?
This is a business call, not a statistical one. It comes down to the cost of rolling the change out (engineering time, infrastructure), the risk involved (user trust, brand), and the expected benefit at scale (revenue, engagement). A feature that takes three months to build probably only earns its keep at a 5%+ lift. A one-day CSS tweak might be worth shipping for a 0.5% lift.
Netflix only needs 476 people per group, since a 9-point lift is easy to spot. Spotify needs 1,512 for a 5-point lift. Google needs 20,330 per group just to catch a 0.5-point CTR lift, and that's exactly why Google can afford to run CTR experiments at all: at billions of searches a day, 20,330 per group happens in minutes. A startup with 1,000 daily users couldn't run this particular experiment no matter how badly it wanted to.
The lesson underneath all of this: the smaller the effect being chased, the more data it takes, and how much data is actually available quietly decides which experiments are even possible to run.
Detecting a 1-point lift takes roughly 234 times more data than detecting a 15-point lift. That's not a rounding curiosity, it comes straight from the shape of the formula. The denominator is (p₂−p₁)², the effect size squared. Halve the effect being chased and the required sample doesn't double, it roughly quadruples, because the effect sits in the denominator squared. This is exactly why teams almost always negotiate the MDE before touching the sample size formula at all, agreeing on the smallest lift worth acting on is really agreeing on how long everyone's willing to wait.
Random assignment is what separates A/B testing from just watching people and drawing conclusions. Get this part wrong and everything downstream is invalid, no matter how careful the analysis looks afterward.
User-level hashing is the standard approach:
MD5 gives three things at once: the same user always lands in the same variant (deterministic), the assignment is uniform and unpredictable (nobody's biased toward being an early or late signup), and different experiments assign the same user differently, since the experiment name is baked directly into the hash.
Before analyzing any results, check that the randomization actually worked. The two groups should look identical on any metric that wasn't being tested.
Typical columns to check: user age (days since signup), historical purchase rate, device type, geography. If any of these come back with p<0.05 for a metric that wasn't even being tested, the randomization has a bug somewhere.
Before trusting any new experiment infrastructure, run an A/A test: both groups get the exact same experience. Run the usual statistical test once the target sample size is reached. What should show up: a p-value that's roughly uniformly distributed (about 5% of A/A tests will come back "significant" purely by chance), and no systematic difference in metrics between the two groups. If A/A tests keep coming back significant, something's actually broken in the randomization or the logging pipeline, not in the world.
This is probably the single most common source of false positives in real-world A/B testing. Understanding it properly is what separates a rigorous data scientist from a careless one.
An experiment is running. The dashboard shows p=0.07. Two days later, p=0.04. It gets called a win and shipped.
This feels completely rational. It's actually statistically disastrous.
Here's why: a p-value is only valid at a single, pre-decided sample size. Every extra time it gets checked, the chance of stumbling into a false positive goes up, even when H₀ is true and there's no real effect anywhere.
Before the code: 10,000 experiments get simulated where it's already known, by construction, that there's no real difference between control and treatment, both groups get the same underlying 50% rate. Since there's genuinely nothing to find, any experiment that declares a "significant" result is by definition wrong, a false positive. For each of those 10,000 fake experiments, two approaches get tried: check the p-value every 50 users and stop the moment it looks significant (peeking), versus check it exactly once at the end (correct). Then just count how often each approach cries wolf.
Peeking pushes the false positive rate from 5% up to 26%. More than one in four experiments declared "winners" this way are actually pure noise. That's shipping things that don't work, over and over, without ever knowing it.
Think of flipping a fair coin. Commit to exactly 100 flips and check significance once at the end, and the wrong conclusion of bias happens about 5% of the time. But check after every single flip and stop the moment p dips under 0.05, and the wrong conclusion happens far more often, because any random little run of heads looks significant if the check stops at exactly the right instant.
The discipline here is simple, even if it takes willpower: calculate the required sample size before starting, set a hard end date or user count, don't look at p-values until that threshold is hit, then analyze once and make one decision.
If an organization genuinely needs to monitor an experiment for safety (a guardrail metric tanking, say), sequential testing methods exist for exactly this, the Sequential Probability Ratio Test or alpha-spending functions, specifically built to allow peeking without inflating Type I error. They're beyond what this article covers, but worth knowing they exist.
Show people something new and they engage with it more, not necessarily because it's better, just because it's unfamiliar and interesting. That effect fades.
Say Swiggy launches a new "food mood" discovery feature. Week one, engagement is 40% higher. Two months later, it's 3% higher. If the A/B test only ran for one week, it would have confidently declared a winner that barely existed.
The fix: run experiments for at least two weeks, four for major UI changes, and plot the treatment effect over time. If it's driven by novelty, the decay will show up on the plot. A genuine improvement holds its lift.
SUTVA (Stable Unit Treatment Value Assumption) is the quiet assumption every A/B test leans on: the treatment one user experiences shouldn't affect another user's outcome.
This breaks down constantly in social or marketplace products. At Uber, if the new UI makes more drivers accept surge trips, there are simply more cars on the road, so control-group riders get better service too, which means the real treatment effect ends up underestimated. At Spotify, a treatment user sharing a new playlist feature with a control-group friend contaminates that friend's experience. On a messaging app, basically any feature involving communication between people can't be cleanly isolated at all.
The fix: cluster randomization, assign whole clusters (friend groups, geographic regions, driver pools) to control or treatment, rather than individuals. This keeps the SUTVA assumption intact at the cluster level, even if it doesn't hold perfectly at the individual one.
A trend that shows up in every subgroup can flip entirely once those subgroups get combined, purely because the groups aren't the same size.
Randomization "worked" in the sense that overall group sizes came out balanced, but the mix inside each group wasn't. The fix: always break results down by the major segments, device type, geography, user tenure, as part of the sanity check. If a segment turns out imbalanced, either stratify the randomization (assign within each segment separately) or control for it statistically with regression.
Track 20 metrics and call something a winner the moment any one of them dips under p<0.05, and the real overall false positive rate sits around 64%, the same math from Day 12. Define the primary metric before the experiment starts. Anything interesting that shows up in a secondary metric is a hypothesis worth testing next time, not evidence already in hand.
Time to run all five experiments start to finish.
Setup: control 2,080/4,000 (52.0%), treatment 2,440/4,000 (61.0%), next-episode rate.
Ship it. z=8.12 with a p-value that's essentially zero. The 9-percentage-point lift (52% to 61%) is both statistically and practically significant, and the CI says the true lift is above 6.8pp even in the pessimistic case. A hypothetical guardrail check on unsubscribe rate and complaints held steady in this scenario. The one open question left is product judgment, not statistics: does a 10-second countdown fit a brand built around user choice and control? That's a call for leadership, not for the p-value.
Setup: driver acceptance rate, simulated for both groups, n=3,500 each, control centered at 68%, treatment centered at 74%.
Ship it. Cohen's d=0.34 is a medium effect, genuinely strong for a pure UI change. Nearly 9,900 extra surge trips accepted per day in one city, with no change to driver pay at all. The CI of roughly 4.3 to 5.6 percentage points gives the product team a defensible range to actually plan around.
Quick reminder from Day 12, since the exact numbers are easy to forget: Cohen's d of roughly 0.2 counts as small, 0.5 as medium, 0.8 as large. These are the thresholds the code's if/else chain checks against.
Setup: control 3,550/5,000 (71.0%), treatment 3,800/5,000 (76.0%), order completion.
Ship it. Five percentage points on order completion across three million daily sessions works out to roughly 150,000 extra orders a day, so this change pays for itself within days. The one-tailed test is defensible here because the direction was genuinely pre-specified from offline evaluation, and a worse result would have triggered the exact same action (rollback) as no result at all.
Setup: control 2,280/6,000 (38.0%), treatment 2,580/6,000 (43.0%), 30-day retention.
Ship it. Five percentage points of 30-day retention is a 13.2% relative lift, worth roughly $300K in extra monthly recurring revenue from just one month's cohort, and that compounds as more cohorts stack up. The two-tailed test was the right call here too, an unfamiliar interactive onboarding flow could plausibly have confused new users and hurt retention, and this test design would have caught that just as readily as it caught the improvement.
Setup: control 2,480/80,000 (3.1%), treatment 2,880/80,000 (3.6%), CTR on result 1.
Cohen's h is used here instead of Cohen's d, and that's not a typo, proportions genuinely need a different effect size measure. Cohen's d assumes a difference can be standardized by dividing by a standard deviation that stays roughly constant. But a proportion's variance is p(1−p), which changes depending on where p sits, the variance near 50% is much bigger than the variance near 3% or 97%. A 0.5pp difference near a 50% baseline and a 0.5pp difference near a 3% baseline don't represent the same amount of real change, dividing by the raw standard deviation would understate how big the second one actually is. Cohen's h fixes this with an arcsine transformation that stretches out the space near 0% and 100%, so differences become comparable no matter where the baseline sits. The same small/medium/large thresholds (0.2/0.5/0.8) apply, just on this transformed scale.
Ship it. This is the statistical-versus-practical-significance lesson running in reverse. Cohen's h=0.028 is tiny by any conventional yardstick. But at roughly 8.5 billion searches a day, that 0.5pp lift works out to about 42 million extra clicks daily, or somewhere around 15.5 billion a year. An effect size means nothing on its own, it only means something once the denominator it's scaling against is known.
Classical A/B testing treats the whole thing as pure learning, accepting some cost during the experiment in exchange for knowledge, then acting only once it's over. A multi-armed bandit instead shifts traffic toward whichever variant is currently winning while the experiment is still running, cutting down that cost.
Picture a casino with three slot machines. Classical A/B testing says: play each one the same number of times, then pick the best. A bandit says: pull each one a few times, then start playing the best-performing one more and more, adjusting continuously.
Bandits earn their keep when the experiment is long-running and leaving traffic on the losing variant isn't affordable, when the effect is large and detectable quickly, or when personalizing and different users might have genuinely different "best" variants. The trade-off is real though: bandits sacrifice some statistical power and are harder to reason about than a clean A/B test. For major product decisions, a fixed-sample A/B test is still the gold standard.
Sequential testing (SPRT, the Sequential Probability Ratio Test) allows looking at the data as it accumulates and stopping the moment there's enough evidence, without inflating Type I error. This is the actual, formal fix for the peeking problem, checking as often as needed becomes fine, because the stopping rule is built directly into the statistical test itself, not bolted on afterward. It's increasingly common at companies like Airbnb and Spotify for long-running engagement experiments.
Before the experiment: define one primary metric that decides ship/no-ship. Define guardrail metrics that must not degrade. State H₀ and H₁, choose one- or two-tailed. Set α=0.05, target power=80%. Decide the MDE, the minimum lift actually worth caring about. Calculate the required n per group. Determine experiment duration, minimum 1 to 2 weeks.
During the experiment: hash-based random assignment, so the same user always gets the same variant. Do not peek at p-values. Monitor guardrail metrics for safety only.
After the experiment: sanity check, are baseline metrics balanced across groups? Run the correct test (proportion z or Welch's t). Report p-value, effect size, 95% CI, and absolute lift together. Segment the analysis (device, region, user tenure). Check guardrail metrics. Make one decision: ship, no-ship, or iterate.
Common traps: peeking and stopping early (false positives jump from 5% to 26%). Changing the primary metric after seeing results (p-hacking). Ignoring the novelty effect (first-week enthusiasm). Individual randomization in social or marketplace products (SUTVA violations). Running 20 metrics at α=0.05 (a 64% chance of at least one false positive). Large n making tiny, meaningless lifts "significant." Before-after comparisons instead of a parallel control group.
A/B testing is hypothesis testing with a control group. The random split is what actually makes causal inference possible, without it it's just correlation analysis wearing an experiment's clothes. The most important work happens before the experiment starts, defining the primary metric and computing the required sample size in advance is what separates a rigorous experiment from a fishing expedition. MDE drives everything, sample size scales with the inverse square of the effect being chased, so halving the MDE roughly quadruples the data needed, and how much traffic is actually available quietly decides which experiments are even possible. Peeking destroys experiments, the simulation showed the false positive rate jumping from 5% to 26%, fix the sample size, set the end date, and don't look until it's actually done. Statistical significance alone isn't enough, Google's 0.5pp CTR lift had a tiny effect size and an enormous business impact, the p-value says the effect is real, effect size and scale say whether it matters. Guardrails stop a battle win from becoming a war loss, autoplay could raise next-episode rates while quietly raising unsubscribes too. And sanity checks come before analysis, always, verify that randomization actually produced balanced groups on pre-experiment metrics, since groups that differ going in make anything found coming out confounded.
Thirteen days ago, this series opened with a simple question: how does a data scientist think about numbers? Everything since has built toward this exact article. Descriptive statistics gave the vocabulary. Probability and distributions gave the machinery. The Central Limit Theorem made confidence intervals possible. Hypothesis testing turned those intervals into decisions. And A/B testing is where all of it lands: the actual process a real company runs, this week, to decide whether to ship something.
The formulas were never really the point. The point was always this: a sample can't be trusted blindly, and it can't be dismissed blindly either. Somewhere in between sits an honest, quantified answer, and now there's a full toolkit for finding it.
Interview-style questions on A/B testing, the kind that actually come up at places like Google, Meta, Airbnb, and Uber.
The MDE is the smallest effect that would be practically meaningful enough to actually ship the change. It's a business decision, not a statistical one, set before computing a sample size, not after. Three things shape it. Cost: a feature needing six months of engineering demands a real lift to justify it, maybe MDE=5pp, while a one-day CSS change might be worth shipping for 0.5pp. Risk: changes to core experience like onboarding or payment need a meaningful MDE, since even a small negative effect there is a real risk, while a low-risk backend change can tolerate a smaller MDE. Scale: at Google's volume, even a 0.1pp CTR MDE is worth chasing, while at a startup with 10K daily users, a 0.1pp lift is both too small to measure reliably and too small to care about. The practical move: get the PM, the engineering lead, and the data scientist to agree on the MDE before anyone writes the sample size formula, that conversation also quietly forces everyone to agree on what "success" actually means.
Six things, roughly in this order. Did the primary metric actually pass, or was this a secondary metric? A secondary metric coming in significant is a hypothesis worth testing next time, not a ship signal today. What's the effect size and absolute lift? p=0.03 says the effect is real, not that it matters, a 0.01pp lift with a million users clears p<0.05 easily and still isn't worth shipping. Are the guardrails intact? A 2pp conversion lift alongside a 1pp jump in unsubscribes could easily be a net negative. Did the test actually reach its planned sample size? If it was stopped early, the result might just be a peeking artifact. Any segment imbalance hiding underneath, an overall lift masking real harm to a specific segment, exactly the Simpson's paradox risk? And could this be novelty? If it's week one of a major UI change, wait for week two before trusting the number.
The randomization unit is what actually gets assigned (user, session, device). The analysis unit is what gets measured and compared (a click, an order, a conversion event). When they don't match, pseudo-replication shows up, multiple correlated measurements from the same unit get counted as if they were independent observations. That violates the independence assumption baked into the test, understates the real variance, inflates the test statistic, and produces false positives. Randomizing by user but measuring CTR by click event means a user who searches 50 times contributes 50 "observations," but they're not independent, they all come from one person with consistent behavior. The standard error ends up too small, the test statistic too large, and the p-value looks far more confident than it should. The fix: aggregate up to the randomization unit before testing. If randomized by user, compute each user's own CTR first and compare those user-level averages across groups. n becomes the number of users, not the number of clicks.
Both show up in A/B tests for proportions, and which one to use is a principled choice, not a stylistic one.
This is the same distinction Day 12 made for single-proportion tests, where p₀ was used inside the test but p̂ inside the CI. Using pooled SE for a CI, or unpooled SE for a test statistic, is technically wrong either way, the results only drift a little for large n, but using the right one for each calculation is exactly the kind of precision that separates careful work from sloppy work.
Extending an experiment after seeing a significant result is just as risky as stopping early, it's p-hacking wearing a different outfit. If the extension is hoping to "confirm" a significant result with more data, that's now a second, informal test with stopping rules that were never actually pre-specified, and the Type I error rate is no longer under control at α. If instead the extension happens because the result was not significant and more data is hoped to get there, that's textbook p-hacking, fishing until the number finally cooperates. The right answer: fix the duration and sample size before the experiment begins. Follow that discipline, and the result at the pre-specified endpoint is valid on its own, no extension required. The one legitimate reason to extend: "the target sample size hasn't been hit yet because traffic came in lower than expected." That's not fishing for significance, that's just finishing what was planned.
Two distinct risks. Interaction effects: if experiment A changes the homepage and experiment B changes checkout, users exposed to both might behave differently than users exposed to just one, or neither. What gets measured for each experiment is really the effect conditional on the other being present, not the clean, isolated effect it was supposed to be. Statistical collision: if the same users sit in A's treatment and B's control, and the two changes push the same metric in opposite directions, both effects end up underestimated at once. Mitigations, roughly cheapest to most robust: traffic isolation, give each concurrent experiment its own non-overlapping slice of users, clean but wasteful with limited traffic. Factorial design, explicitly assign users to one of four cells, (A_ctrl, B_ctrl), (A_trt, B_ctrl), (A_ctrl, B_trt), (A_trt, B_trt), then analyze with two-way ANOVA, running one experiment with four groups instead of two separate 50/50 experiments, which reveals not just what A does and what B does, but whether having both at once produces something different from just adding their effects together. Sequencing, just run the experiments one after another, cleaner but slower. Companies running thousands of simultaneous experiments build dedicated overlapping-experiment infrastructure to handle this automatically. Most other organizations should default to traffic isolation until their tooling matures enough to do better.
A handful of explanations, each pointing at a different failure mode. Novelty decayed, the test captured a week or two of peak newness, and the lift quietly evaporated once things stopped feeling new. The holdout wasn't representative, test on 5% of users and the other 95% might simply differ in demographics or usage patterns once rolled out fully. The metric wasn't the real goal, optimizing for click rate when the actual business goal was revenue per session, and the two can move in opposite directions. Network effects weren't captured, in a marketplace, scaling a change up alters market dynamics (more demand, fewer available drivers, longer waits) in ways a small test could never have shown. Or it was a false positive all along, at α=0.05, one in twenty genuinely valid experiments will declare a winner that isn't real. The lesson: A/B tests are necessary, but not sufficient on their own. Always run post-launch monitoring for at least four weeks to confirm the result actually holds up at full traffic.
The formula doesn't change, but rare-event metrics need enormous samples, exactly why they tend to make poor primary metrics for A/B tests. Sizing a test to detect a move from 0.5% to 0.4% failure needs roughly 70,000+ per group. A few ways around this when a rare metric is genuinely unavoidable: find a proxy metric that's more frequent and correlates with the rare one ("payment retry attempted" might stand in for "payment failed"). Use a ratio metric measured within a smaller, high-volume segment where the failures actually concentrate. Increase traffic allocation, putting 50% of all traffic into the experiment instead of 10% gets 5× the sample in the same calendar time. Or just accept a longer runtime, sometimes the honest answer really is that detecting a meaningful change in a rare event takes more time, full stop.
A guardrail metric is one that must not get worse during an experiment, no matter how well the primary metric performs. It's the floor a team won't ship below, regardless of the headline result. Netflix's primary metric might be next-episode click rate, with unsubscribe rate as the guardrail. A feature that doubles next-episode clicks while also raising unsubscribes by 50% isn't a win, it's extraction. Can a guardrail ever be knowingly crossed? Yes, but only as an explicit, documented leadership decision, never as an analyst's unilateral call. Legitimate cases exist: the degradation is small and sits inside a pre-agreed tolerance band, the primary gain is large enough to clearly justify the trade, or the degradation is expected to be temporary, a learning-curve effect during a new onboarding flow, say. In these cases, the data scientist's job is to lay the trade-off out plainly, primary improved by X, guardrail degraded by Y, and let the product and leadership team make the ship or no-ship call with full information in front of them. Deciding on one's own that a guardrail breach is "fine" is never the analyst's decision to make.