Frequentist Hypothesis Testing:
In **frequentist** hypothesis testing, we begin with the **null hypothesis** (H0), which often represents no effect or no difference (e.g., “A is equivalent to B”). We assume H0 is true and gather data to see whether the observed evidence is sufficiently unlikely under H0. Our goal is **not** to prove H0 true; rather, we want to see if we can **reject** H0 in favor of the **alternative hypothesis** (Ha), which posits a difference or effect (e.g., “A is not B”).
We do not aim to confirm or “prove” H0; instead, we attempt to falsify it. **Failing to reject** H0 does **not** mean H0 is true—it only indicates that the data do not provide sufficient evidence to discard H0.
The decision to reject or fail to reject H0 typically depends on the **p-value**, which is the probability of observing data at least as extreme as what we observed, given H0. A **small p-value** (often < 0.05) suggests that such data would be unlikely if H0 were true, so we **reject** H0 in favor of Ha. A **larger p-value** indicates insufficient evidence to reject H0.
1. **Assume** H0 is true.
2. **Seek evidence** that would be unlikely under H0 (this underlies the p-value calculation).
3. **Reject** H0 if the evidence is strong (i.e., p-value < your chosen significance level), providing support for Ha.
4. If we **cannot reject** H0, we are **not** proving it; we simply lack sufficient evidence to falsify it.
A Note on Bayesian Methods:
By contrast, in **Bayesian** inference, probability reflects a *degree of belief*. Bayesian methods incorporate **prior** beliefs about parameters or hypotheses and then **update** those beliefs as data are observed, yielding a **posterior** probability of the hypothesis itself. Whereas frequentist methods focus on the probability of the **data** given H0, Bayesian approaches directly quantify the probability of H0 (or other hypotheses), given the data and prior. This leads to different ways of interpreting evidence but does not negate the utility of frequentist p-values as a widely used decision criterion.
Example
Simple A/B (Headache Medication) Test:
Imagine you’ve developed a new headache medication. You gather 100 volunteers with chronic headaches and randomly assign 50 to receive the new medication and 50 to receive a placebo (sugar pill).
- **Null Hypothesis (H0):** The new medication has **no** effect; pain relief in the treatment group is no different from that in the placebo group.
- **Alternative Hypothesis (Ha):** The new medication **does** relieve headaches more effectively than a placebo.
You measure each volunteer’s pain score after treatment and compute a p-value—i.e., the probability of observing such a difference (or a bigger one) **if** the new medication was no better than placebo.
- If the **p-value** is very low (e.g., < 0.05), you **reject** H0, suggesting the medication is likely more effective than the placebo.
- If the **p-value** is not below your threshold, you **fail to reject** H0. This doesn’t prove the medication is ineffective; it only means you lack enough evidence to conclude otherwise.
Two Other Notable Testing Methods:
- **Multi-Armed Bandit**: Dynamic, adaptive allocation emphasizing regret minimization and continuous learning, rather than a single end-of-test analysis.
- **Multivariate/Orthogonal Array**: Systematic testing of multiple variables/factors at once, leveraging experimental design principles to detect main and interaction effects more efficiently than multiple separate A/B tests.