Day 11 answered "what is the value?" This one answers a harder question: is what we're seeing real, or just noise? That question is the engine behind every A/B test, every product experiment, and pretty much every data-driven decision at any tech company you can name.
In Day 11, Spotify sampled 10,000 users and built a 95% CI for their conversion rate: [11.36%, 12.64%]. That told us where the true rate probably sits.
Now say Spotify's growth team redesigns the Premium upgrade page and runs it for a sample of users. They see a new conversion rate of 13.5%. The question isn't "where is the rate?" anymore. It's: "is 13.5% genuinely better than our baseline of 12%, or did we just get lucky with this sample?"
That's a hypothesis test. And answering it properly, without quietly fooling yourself, is one of the most useful skills in data science.
Here are five problems from the same five companies as before, now reframed as hypothesis testing questions.
Netflix's engineering team ships a new recommendation engine. The old algorithm produced an average watch time of 108 minutes a day. They run the new algorithm on a sample of 1,200 users and see an average of 112 minutes a day, with a standard deviation of 47 minutes.
The question: "Is that 4-minute improvement real, or could it just be random sampling variation? Should we roll this out to all 260 million users?"
Uber's maps team rolls out a new routing algorithm in Mumbai. Historically, average trip duration has been 22 minutes. A sample of 800 trips under the new algorithm shows an average of 20.3 minutes, with a standard deviation of 11 minutes.
The question: "Is a 1.7-minute improvement actually statistically significant, or is that just something you'd see by chance in any random sample of 800 trips?"
Swiggy's operations team rolls out a new delivery partner incentive scheme. They sample 400 deliveries under the new scheme. The sample average is 28.4 minutes, with a standard deviation of 6.8 minutes.
The question: "Can we formally claim the true average delivery time is now under 30 minutes, or is this sample not strong enough evidence to say that?"
Spotify redesigns their Premium upgrade page. The old version converted at 12%. They show the new page to 8,000 free users and see 1,080 conversions, that's 13.5%.
The question: "Is 13.5% actually higher than 12% in a meaningful way? Did the redesign genuinely work, or is this just a fluke?"
YouTube's product team wants to know whether mobile and desktop users consume content differently. They sample 600 mobile users (average 42 minutes a day, std 18) and 500 desktop users (average 38 minutes a day, std 22).
The question: "Is that 4-minute gap between mobile and desktop users real, or is it within the range of ordinary random noise? Should the product team be building separate content strategies for each platform?"
Every one of these is really the same core question wearing a different outfit: is the pattern seen in the sample strong enough to believe it holds for the whole population? Let's build a framework that answers that properly, every time.
Hypothesis testing borrows its logic straight from a courtroom.
In a trial, the defendant is presumed innocent until proven guilty. The prosecution has to bring evidence strong enough to overturn that presumption. The jury never asks "is the defendant innocent?" They ask "does the evidence cross the bar needed for conviction?"
Hypothesis testing works exactly the same way. The null hypothesis (H₀) is the presumption of innocence, the boring, conservative claim that nothing changed and nothing worked. The alternate hypothesis (H₁) is the prosecution's claim, that the new thing worked, that the difference is real. The data, the sample, is the evidence. The significance threshold (α) is how much evidence counts as proof. The p-value shows how strong the evidence actually is against the null.
H₁ never gets "proven" true. The only question is whether the evidence is strong enough to reject H₀, the same way a jury never declares someone innocent, only "not guilty," meaning there wasn't enough evidence to convict.
Think of it like being a quality inspector at Swiggy. The default assumption is that delivery times are fine (H₀: average ≥ 30 minutes, no real improvement). The new incentive scheme is on trial. 400 deliveries get collected, and the question becomes: if the scheme actually did nothing, how likely is it a sample average this low would show up just by chance? If that probability turns out tiny, that's compelling evidence, reject H₀ and conclude the scheme genuinely works.
Every hypothesis test, no matter what kind, follows the same five steps. Learn this once and it applies to any test you'll ever run into.
Null hypothesis (H₀): the default, conservative claim. Always contains an equality (=, ≤, or ≥). Always the thing being knocked down.
Alternate hypothesis (H₁): the claim actually being supported with evidence. What's believed true if H₀ turns out to be wrong.
| Business question | H₀ | H₁ |
|---|---|---|
| Did the Netflix algorithm improve watch time? | μnew ≤ 108 min | μnew > 108 min |
| Is Swiggy's delivery time under 30 min? | μ ≥ 30 min | μ < 30 min |
| Did Spotify's redesign improve conversion? | pnew ≤ 0.12 | pnew > 0.12 |
| Is YouTube mobile ≠ desktop watch time? | μmobile = μdesktop | μmobile ≠ μdesktop |
One-tailed vs. two-tailed tests. A one-tailed test has H₁ pointing in a specific direction, greater than or less than. Use this when only improvement, or only decline, matters. Netflix, Uber, Swiggy, and Spotify above are all one-tailed. A two-tailed test has H₁ just saying "different from," with no direction specified. Use this when a difference in either direction would matter. YouTube above is two-tailed.
One-tailed is a bit like asking "did the new medicine make patients better?" Two-tailed is asking "did the new medicine change patient outcomes at all?" The second question catches improvements and unexpected harm, both.
The significance level (α) is the probability of making a wrong rejection, concluding an effect is real when it actually isn't. It's the threshold that decides what counts as strong enough evidence.
| α | Equivalent confidence | Typical use |
|---|---|---|
| 0.10 | 90% | Exploratory analysis, low-stakes decisions |
| 0.05 | 95% | Standard in most data science and research |
| 0.01 | 99% | High-stakes decisions (medical, financial, legal) |
α gets chosen before looking at the data. Changing it after seeing the results is called p-hacking, one of the more common, more dangerous forms of statistical malpractice out there.
It helps to think of α like setting the conviction threshold before the trial even starts. A threshold of α=0.05 basically means: "we'll only convict if there's less than a 5% chance this evidence would show up when the defendant is actually innocent." Changing that threshold after already seeing the evidence is just rigging the trial.
The test statistic turns a sample result into a standardized score, showing how many standard errors away from the null hypothesis the observation actually sits.
That "df = n − 1" is the degrees of freedom, worth understanding rather than just memorizing. Once the sample mean is calculated from n data points, only n−1 of them are actually free to vary, the last one is locked in, because all n values together have to average out to x̄. Fewer degrees of freedom means the t-distribution has fatter tails than the normal curve, exactly why small samples need a wider critical value to reach the same confidence. As n grows, df grows with it, and the t-distribution quietly turns into the z-distribution, the whole reason the two barely differ once n is large.
For two-sample means:
For proportions:
It's a bit like converting someone's height into a z-score. Instead of saying "this person is 185 cm," the statement becomes "this person is 1.8 standard deviations above average." It puts every result on the same scale, so completely different problems become comparable side by side.
The p-value is the probability of seeing a test statistic as extreme as, or more extreme than, the one actually observed, assuming H₀ is true.
A small p-value is basically saying: "if H₀ were true, seeing this result would be really unlikely, so maybe H₀ isn't true after all."
What the p-value is NOT: it is not the probability that H₀ is true. It is not the probability of having made an error. It is not the probability that the result was due to chance.
What it actually is: the probability of the data, or something more extreme, given H₀ is true. A measure of how surprising the evidence is, assuming the null holds.
Picture flipping a coin suspected of being biased toward heads. H₀ says it's fair. 20 flips produce 17 heads. The p-value asks: if this coin really were fair, what's the probability of getting 17 or more heads out of 20 flips? That works out to about 0.0013, roughly 1 in 800. Genuinely surprising under H₀. That's compelling evidence the coin is biased.
Two language distinctions that matter a lot in practice: "fail to reject H₀" does not mean "H₀ is true." It means there isn't enough evidence to reject it. "Reject H₀" does not mean "H₁ is definitely true." It means the data is inconsistent with H₀ at the chosen significance level.
Every decision in a hypothesis test can be wrong in one of two ways, and understanding both really matters, since they carry different consequences and different industries tolerate them differently.
Rejecting H₀ when it's actually true, concluding the new algorithm works when it really doesn't. Its probability is α, the significance level chosen, and lowering α reduces how often this happens.
Netflix example: concluding the new recommendation engine increased watch time and rolling it out to 260 million users, when in reality the sample difference was pure luck and the algorithm is no better than the old one.
Failing to reject H₀ when it's actually false, missing a real effect that's genuinely there. Its probability is β, controlled by increasing sample size, increasing effect size, or raising α.
Netflix example: the new algorithm genuinely does improve watch time, but the sample was too small to pick it up, so H₀ doesn't get rejected and an algorithm that actually would have helped gets shelved.
Power is the probability of correctly catching a real effect when one actually exists. It's the number worth keeping high.
A test with 80% power (β=0.20) will catch a true effect 80% of the time, and miss it the other 20%. 80% power is the standard minimum across most industries.
A few things push power up: a bigger sample size, the biggest lever available. A bigger true effect size, though this can't be controlled, knowing it helps with planning. A higher α, though this also raises the Type I error rate, a real trade-off. And less noisy data, since cleaner measurement genuinely helps.
Type I error is a fire alarm going off when there's no fire. Type II error is a fire burning while the alarm stays completely silent. α controls the false alarms. Power controls how reliably the alarm actually goes off during real fires. Turning down false alarms (lowering α) tends to make the alarm less sensitive too (lowering power), so both can't be fixed at once without more data.
Different situations call for different test statistics.
| Situation | Comparing | Test |
|---|---|---|
| One sample vs. a known value | a mean to μ₀ | One-sample t-test (or z-test if σ is somehow known) |
| One sample vs. a known value | a proportion to p₀ | One-sample z-test for proportions |
| Two independent samples | two means | Two-sample t-test (Welch's) |
| Same group, before/after | two means | Paired t-test |
| Two independent samples | two proportions | Two-proportion z-test |
One-sample z or t test. Use this comparing a sample mean to a known benchmark, a historical value, an industry standard, an SLA target.
Two-sample t-test (Welch's). Use this comparing means from two independent groups, control vs. treatment, mobile vs. desktop. Welch's version doesn't assume the two groups have equal variances, so it's almost always the better default over the classical pooled t-test.
Paired t-test. Use this when the same subjects get measured twice, before and after. Pairing removes person-to-person variability, making the test more sensitive to the actual change.
Proportion z-test. Use this when the outcome is binary, converted or didn't, churned or retained, and a rate is being compared to a baseline or between two groups.
This is probably the most under-taught idea in hypothesis testing, and arguably the most important one once actually working in industry.
Statistical significance answers: is the effect real, or could it just be noise? Practical significance answers: is the effect actually big enough to matter? These two things are completely independent of each other.
A result can be statistically significant but practically useless. At large enough n, even a 0.001-minute difference in watch time will have p<0.05. Does 0.001 minutes matter to Netflix? Not even slightly. A result can also be practically meaningful but not statistically significant. A small sample can easily miss a genuine 10-minute improvement in delivery time simply because n wasn't big enough to detect it.
The standard measure for means is Cohen's d:
Dividing by spooled does the same job the z-score analogy did back in Step 3, converting a raw difference into a standardized one, measured in standard deviations instead of whatever units the metric happens to be in. That's what makes Cohen's d comparable across completely different problems: a d of 0.5 means the same thing whether comparing minutes of watch time or dollars of transaction value, in both cases the two groups sit half a standard deviation apart.
| d | Practical significance |
|---|---|
| 0.2 | Small effect |
| 0.5 | Medium effect |
| 0.8 | Large effect |
For proportions, the raw difference (p̂₁ − p̂₂) is usually the most intuitive number to look at. A lift from 12% to 12.1% will have p<0.05 with a million users, but that's not a redesign worth shipping.
Imagine a drug trial shows a new medication lowers blood pressure by 0.1 mmHg with p=0.0001. That's statistically significant, purely because the sample is enormous. But a 0.1 mmHg change is clinically meaningless, smaller than the measurement error on a blood pressure cuff. Statistical significance without practical significance is basically a hollow win.
The rule of thumb in industry: always report the p-value and the effect size together, or the raw difference. Make the actual business call based on both, never just on whether p is under 0.05.
Say YouTube is testing 20 different app changes at the same time, each using α=0.05. What's the chance of getting at least one false positive somewhere across those 20 tests?
Here's where that formula comes from. The chance any single test does not produce a false positive is (1−α)=0.95. If the 20 tests are independent of each other, the chance all 20 avoid a false positive at once is 0.95 multiplied by itself 20 times, 0.95²⁰. Subtract that from 1, and the result is the probability at least one of them slips up.
There's a 64% chance of ending up declaring at least one "winner" that's actually just noise. This is the multiple testing problem, and it's a genuinely major source of false discoveries in industry.
Divide the significance threshold by the number of tests being run:
Running 20 tests and wanting an overall α of 0.05 means αadjusted = 0.05/20 = 0.0025. Now each individual test uses a stricter bar, keeping the overall false positive rate at or below 5%.
One honest caveat worth knowing: that original 64% calculation assumed the 20 tests were independent of each other. In practice, metrics from the same experiment are often correlated, click rate and time-on-page tend to move together, so the real inflation in false positives can sometimes be a bit lower than the formula suggests. Bonferroni doesn't need the tests to be independent to still work as a safe, valid bound, which is part of why it's conservative: it protects against the correlated case too, at the cost of being stricter than strictly necessary in the independent case.
Bonferroni is conservative, it sometimes misses real effects by being too strict. More powerful alternatives exist (Benjamini-Hochberg, FDR control), but Bonferroni is the intuitive place to start, and it's more than enough for most interview conversations.
Time to solve all five business problems with real Python code. Every solution walks through the same five steps.
Setup: old algorithm baseline μ₀=108 min/day. New algorithm tested on n=1,200 users: x̄=112 min, s=47 min. Did it actually improve things?
The test rejects H₀ with p=0.0016, well under 0.05. The 4-minute improvement is statistically real, not a sampling fluke. That said, Cohen's d=0.085 says this is a small effect. Netflix's leadership still has a real question to answer: is a 4-minute increase (a 3.7% lift) worth whatever it costs to run the new algorithm? Statistical significance says the effect is real. It doesn't say whether it's worth it, that's a business judgment call.
Setup: baseline μ₀=22 min. New algorithm on n=800 trips: x̄=20.3 min, s=11 min. Did it actually reduce trip time?
The p-value here is vanishingly small, about 7 in a million. The 1.7-minute reduction is absolutely real. And even though Cohen's d (0.155) is small, the scale of Uber's business turns this into something huge: over 14,000 driver-hours saved every single day, in one city alone. Effect size alone doesn't tell you the whole story in industry. At Uber's scale, even tiny per-trip improvements add up into something enormous.
Setup: Swiggy claims ≤30 minutes. New incentive scheme on n=400 deliveries: x̄=28.4 min, s=6.8 min. Can this claim actually be backed up formally?
Hypothesis testing and the confidence interval from Day 11 tell a consistent story here. The p-value is tiny, and the entire 95% CI, [27.7, 29.1] minutes, sits comfortably below 30. Swiggy's ops team can tell leadership, with real statistical backing, that the new incentive scheme is working. Notice too how the CI gives you more: it shows roughly where the true mean sits, while the hypothesis test just gives a yes-or-no verdict. In practice, report both.
Setup: baseline conversion p₀=12%. New page shown to n=8,000 users: 1,080 conversions, p̂=13.5%. Did the redesign actually work?
For proportions, the formula shifts slightly: p̂ = sample proportion = x/n, SE = √(p₀(1−p₀)/n), z = (p̂−p₀)/SE.
This one's both statistically and practically significant. The redesign delivers a 12.5% relative lift (12% up to 13.5%), and a p-value of 0.000018 is nowhere near any reasonable threshold. At Spotify's scale, 1.5 percentage points works out to roughly 5.25 million additional Premium subscribers, potentially hundreds of millions of dollars in annual recurring revenue. A clear ship decision.
The standard error here uses p₀, the null hypothesis value, not p̂, the sample proportion. That's because under H₀ we're assuming p=p₀ is true, that's the world the probability is being calculated under. This is subtly different from the confidence interval formula back in Day 11, which correctly uses p̂, since no null hypothesis is being assumed there.
Setup: mobile users (n₁=600, x̄₁=42 min, s₁=18 min) versus desktop users (n₂=500, x̄₂=38 min, s₂=22 min). Is that 4-minute difference actually real?
The 4-minute difference is real, p=0.001164. Mobile users genuinely watch more per day than desktop users. The 95% CI, [1.59, 6.41] minutes, says the true difference is likely somewhere between 1.6 and 6.4 minutes, so mobile could be anywhere from modestly to substantially more engaged. Cohen's d=0.201 lands right around the small-to-medium line, and at YouTube's scale, even the low end of that range adds up to billions of cumulative minutes globally. Solid grounds to pursue separate content and recommendation strategies for each platform.
Setup: H₀ is the boring claim (no effect, no change, nothing happened). H₁ is what's being argued for (effect exists, improvement real). α gets set before seeing data, usually 0.05.
Choosing your test: one sample vs. a known value, mean, use a one-sample t-test, t=(x̄−μ₀)/(s/√n). Proportion, use a proportion z-test, z=(p̂−p₀)/√(p₀(1−p₀)/n). Two independent samples: means use Welch's two-sample t-test, proportions use a two-proportion z-test. Same group measured twice: a paired t-test, computing the differences and running a one-sample t-test on those.
Tail direction: "greater than" is right-tailed, p=P(T>tstat). "Less than" is left-tailed, p=P(T<tstat). "Different from" is two-tailed, p=2×P(T>|tstat|).
Decision: p≤α means reject H₀, statistically significant. p>α means fail to reject H₀, not enough evidence.
Always also report: effect size (Cohen's d, or raw lift for proportions), a confidence interval for the effect, and a business interpretation, not just p<0.05.
Watch out for: multiple testing (use a Bonferroni correction), large n making tiny effects "significant," changing α after seeing results (p-hacking), and confusing "fail to reject H₀" with "H₀ is true."
| Day 11: Estimation | Day 12: Decision | |
|---|---|---|
| Question | What is the value of this population parameter? | Is this observed effect real, or is it noise? |
| Path | Sample → CLT → Standard Error → Confidence Interval | H₀ vs H₁ → Test Statistic → p-value → Decision |
| Answer | "The true mean is between X and Y" | "The effect is real / not proven at α=0.05" |
They share the same engine, CLT and Standard Error, and they're complementary: the CI estimates, the test decides.
They're closely related, really two different ways of expressing much of the same underlying calculation, though the exact match depends on lining up the tails correctly. A 95% CI and a two-tailed hypothesis test at α=0.05 will always agree with each other: if the null value (μ₀) sits outside the 95% CI, the test rejects H₀ at α=0.05, and if μ₀ sits inside it, the test fails to reject.
The Swiggy example is a good illustration, with one nuance worth being precise about: Swiggy's test was one-tailed, not two-tailed. Its 95% CI was [27.73, 29.07] minutes, and the null value μ₀=30 sits outside that interval, so the one-tailed test rejects too, that direction of the relationship holds here because clearing the two-sided 95% bar is a strictly harder bar to clear than a one-tailed test at α=0.05 requires. But the reverse isn't guaranteed in general: a one-tailed test can reject in cases where the null value would still fall inside a 95% two-sided CI. For a clean, exact match, pair a two-tailed test with a two-sided CI at the same confidence level, or a one-tailed test with a one-sided CI at that level instead (more on exactly this mismatch in the quick check below).
The practical difference comes down to what each one actually tells you. The CI is richer, showing roughly where the true value lives, genuinely useful for planning. The hypothesis test gives a clean yes-or-no verdict, useful when a ship or no-ship decision is all that's needed. Best practice in industry is to report both, the test gives the decision, the CI gives the size and the uncertainty behind it.
Hypothesis testing is structured decision-making. The five-step framework, state hypotheses, choose α, compute the test statistic, find the p-value, decide, applies to every test you'll ever run. H₀ is always the conservative claim, H₁ never gets proven, evidence just builds up against H₀ until it becomes impossible to defend. The p-value is not the probability that H₀ is true, it's the probability of the data, or something more extreme, given that H₀ is true. A small p-value means the data is surprising if H₀ holds, nothing more. Type I error (α) and Type II error (β) trade off against each other, lowering α cuts false positives but also cuts power to catch real effects, more data is the only lever that improves both at once. Statistical significance is not practical significance, always report effect size right alongside the p-value, since a large enough sample can make a completely meaningless difference look "significant." Multiple testing inflates false positives fast, run 20 tests at α=0.05 and there's a 64% chance of at least one spurious "significant" result, Bonferroni or something similar fixes this. And confidence intervals and hypothesis tests are two sides of the same coin for a two-tailed test: an effect is significant at α=0.05 exactly when the 95% CI for that effect excludes the null value. Report both wherever possible.
Estimation and hypothesis testing together cover most of what a data scientist reaches for day to day: knowing where a value probably sits, and knowing whether a change is real. Both rest on the same foundation this series built from the ground up, starting with descriptive statistics, through probability, through the Central Limit Theorem. From here, that foundation extends into regression, experimentation platforms, and the machine learning models that quietly run hypothesis tests and confidence intervals under the hood every time they report a metric.
Interview-style questions on hypothesis testing, the kind that actually come up at places like Google, Meta, Airbnb, and Uber.
A Type I error, a false positive, is rejecting H₀ when it's actually true, declaring the new algorithm works when it doesn't. Its probability is α, set directly via the significance threshold. A Type II error, a false negative, is failing to reject H₀ when it's actually false, missing a real improvement that was genuinely there. Its probability is β, and (1−β) is the test's power. Which one's worse depends on the stakes. In medical trials, a Type I error means approving a drug that doesn't work, a Type II error means rejecting a drug that could have saved lives, both severe, which is why medical trials tend to use α=0.01 or lower and demand high power. In tech A/B testing, a Type I error means shipping a feature that doesn't help, wasted but usually reversible, while a Type II error means shelving a feature that would have genuinely helped, for a cheap, low-risk feature that missed opportunity can be worse, which is why some companies use α=0.10 for early-stage experiments. There's no universal "worse" error, a good answer ties the choice of α to what each type of mistake would actually cost.
The difference is in what the alternate hypothesis claims, and that decides which part of the distribution the p-value comes from. A one-tailed test has H₁ pointing in a specific direction, and the p-value comes from one tail only, use this when only an effect going one way matters, a product change that could only help (or only hurt), or an SLA that just needs to stay under a threshold. A two-tailed test has H₁ just saying "different from," the p-value covers both tails and gets doubled compared to a one-tailed test, use this when a difference in either direction would actually matter. One practical warning: one-tailed tests are more powerful, easier to reject H₀ with the same data, which makes them tempting to misuse. Running a one-tailed test because "we were hoping the feature would help," when it could plausibly have hurt too, is a design mistake. Two-tailed tests are the safer default for most industry A/B testing.
p=0.04 is technically below 0.05, so H₀ gets rejected, but a good data scientist doesn't stop there. What's the effect size? p=0.04 says the effect is real, it says nothing about whether it's meaningful, a tiny effect that only reached significance because the sample was huge isn't worth shipping. What's the absolute and relative lift? A conversion bump from 12.00% to 12.01% could easily get p<0.05 with a million users, and almost certainly doesn't justify the engineering cost. Are there subgroup effects hiding underneath, an average improvement masking real harm to one segment? How long did the test actually run, stopping the moment p dipped under 0.05 is a classic mistake, since p-values wander during a live experiment, was the pre-planned sample size actually reached? Could this be a novelty effect, people responding well to anything new just because it's new? And were multiple tests corrected for, if this was one of fifteen experiments running at once, a Bonferroni-adjusted threshold would be α=0.05/15=0.0033, and p=0.04 wouldn't come close to significant. A good data scientist lays the evidence out honestly, not just confirms whatever people already want to hear.
P-hacking, sometimes called data dredging, is manipulating an analysis until landing on p<0.05, without a plan set beforehand. It produces false discoveries that don't replicate later. Common ways it happens: running a test over and over and stopping the moment p dips under 0.05 (peeking), trying several statistical tests on the same data and only reporting the one that worked, splitting data into subgroups until one shows significance, swapping the outcome metric after seeing results, or adjusting α after already looking at the data. Prevention: pre-register the experiment, writing down H₀, H₁, α, the primary metric, and the sample size before touching the data. Fix the sample size up front using a power calculation, and don't stop early. Pick a single primary metric, everything else stays exploratory. Apply a multiple testing correction when looking at several subgroups. And keep exploratory analysis separate from confirmatory analysis, anything found while exploring needs validation in a fresh experiment before being treated as real evidence.
It shrinks, gets more "significant," even though the underlying effect hasn't changed at all. The test statistic is t=(x̄−μ₀)/(s/√n). As n grows, the denominator (SE=s/√n) shrinks, making t bigger even if x̄−μ₀ stays exactly the same. A bigger t means a smaller p-value. This is exactly why statistical significance isn't the same as practical significance, given a big enough sample, p<0.05 shows up eventually for basically any non-zero effect, including ones that don't matter in practice. If Netflix had sampled 10 million users instead of 1,200 and still seen the same 4-minute improvement, the p-value would come out essentially zero, but Cohen's d would still be 0.085, still a small effect. The p-value shrank. The actual size of the effect never moved. The lesson: always pair a p-value with an effect size.
The classical two-sample t-test (Student's) assumes both groups share the same population variance, rarely verifiable and often violated in practice. Welch's t-test drops that assumption, adjusting both the test statistic and the degrees of freedom (via the Welch-Satterthwaite approximation) to handle unequal variances properly. That approximation does something fairly intuitive: when the two groups have very different variances, simply adding up the sample sizes and subtracting 2, the way the classical formula does, would overstate how much is actually known. The Welch-Satterthwaite formula shrinks the degrees of freedom to reflect the extra uncertainty from not knowing the variances are equal, exactly why the YouTube example ended up with df=961.7 instead of the clean, classical n₁+n₂−2=1,098. When the variances actually are equal, Welch's barely loses any power. When they're not, it's meaningfully more accurate. The practical rule: just always use Welch's for two-sample comparisons, there's essentially no downside when variances happen to be equal, and it's meaningfully better when they're not.
Pretty much everything. P-values can't be compared like that, as a ratio. p=0.001 and p=0.003 are both comfortably under α=0.05, so both reject H₀, but the gap between them isn't "three times more significant." Statistical significance is a binary decision, reject or don't, not a sliding scale of how correct the conclusion is. On top of that, p-values don't measure effect size at all. p=0.001 could come from a trivial, meaningless difference in a study with ten million people. p=0.03 could come from a genuinely huge, important effect in a small study. For effect size, the right tool is Cohen's d for means or relative lift for proportions. For confidence about the range, the right tool is a confidence interval. A p-value's only job is the reject-or-not decision.
Design, before any data gets collected: define the primary metric, the one number that decides success, plus guardrail metrics that must not get worse. State H₀ and H₁ explicitly, one-tailed or two-tailed depending on whether the change could only help, or could go either way. Set α (usually 0.05) and desired power (usually 80%). Estimate the minimum detectable effect, the smallest improvement actually worth caring about. Run a power calculation for the required sample size per group. Decide how long the test needs to run, long enough to hit the target n and cover at least one full weekly cycle, avoiding day-of-week effects.
Running it: randomly assign users to control and treatment. Don't peek at results to decide when to stop, stop once the pre-determined n is actually reached. Watch for data quality issues, logging errors, unexpected imbalance in assignment.
Analysis: confirm sample sizes match expectations, assignment is balanced, and baseline metrics look similar between groups. Run the right hypothesis test, a proportion z-test for conversion, a t-test for continuous metrics. Report the p-value, the effect size, the 95% CI, and both absolute and relative lift. Check for subgroup effects, and apply multiple testing corrections if needed. Make the actual call: ship it, don't ship it, or run a follow-up experiment.
In theory, this shouldn't happen. A two-tailed test at α=0.05 rejecting H₀ (p<0.05) is mathematically the same thing as a 95% CI excluding the null value. If they disagree, something has gone wrong. Usual suspects: mismatched test types, running a one-tailed test (p from one tail) but reporting a two-tailed CI. A one-tailed test's p-value at α=0.05 actually corresponds to a 90% CI, not a 95% one, so the fix is to match tails, a one-tailed test with a 90% one-sided CI, or a two-tailed test with a 95% two-sided CI. Different formulas could also be the culprit, a z-test for the p-value paired with a t-distribution for the CI, or mismatched standard error formulas somewhere along the way. Or simply a computational slip, worth double-checking that the right functions were used with matching parameters throughout. What to actually do: go back to the raw numbers and recompute both the test and the CI from scratch, using the same formula and the same assumptions throughout. Never publish a p-value and a CI that contradict each other, it signals a real error in the analysis, and any careful reviewer will catch it immediately.