Hypothesis Testing

TODO: Finish this.

p-Value

The null means “The World is as it is.” No associations, no changes. If you want to make a falsifiable claim about The World (and thereby perturb it), a p-value is as easy as this:

What is the probability of seeing what I saw in my experiment if the null hypothesis is true?

78%? Well that sounds bad. You fail to reject the null. 5%? That’s small. Maybe something’s going on? 0.1%? Okay maybe something’s really going on. “Something” here means association, not causation.

Hark!

You never accept the Alternative/Research Hypothesis $H_a$! Falsifiability FTW! You either reject or fail to reject the Null Hypothesis $H_0$.

---

Do you ever set $H_0: \mu \ne \mu_0$ … ?

---

”Which Test?” TLDR (finish this!)

To pick a test, and generally speaking, you’ll be asking

• What is the nature of my Data¹? Continuous? Categorical?
• How many groups am I dealing with? One, two, or more than two?

Here’s a nice little table from this excellent video (by a Columbia alum!)

	1 Group	2 Groups	2+ Groups
Categorical Data	Proportion Test ($Z$-test approx.) $\chi^2$ Test	Proportion Test ($Z$-test approx.) $\chi^2$ Test	$\chi^2$ Test
Continuous Data	$Z$-test & Variants $t$-test & Variants	$Z$-test & Variants $t$-test & Variants	ANOVA ($F$-test, 1-way, 2-way)
Classic Assumptions Violated2	Sign Test Signed Rank Test	Wilcoxon–Mann–Whitney Test Paired $t$-test McNemar’s Test	Kruskal–Wallis Test

Footnotes

1. Always seek to understand your Data all the time 🙏 ↩

2. Too many outliers, small sample size, correlated observations ↩