Inference

So we’ve randomized. We do the experiment. We may see something interesting¹. How do we know this is not due to pure chance? Due to some other funky stuff going on?

Pre-Flight

Fisher tests, Neyman estimates

The 10,000ft overview is:

Question	What	Who
Did it do anything?	Hypothesis Testing	Fisher
How much did it do?	Effect Estimation	Neyman

A Reminder on The Null

The Null hypothesis for non-sophisticated people is: Nothing Happened. There was no effect. The average mean was zero. The ratio was one. And so on. The World is as it is before and after your intervention. You start with this, a most conservative position.

You also never ‘prove’ the Null or your Alternate hypothesis. You either reject or fail to reject the Null. Double-negatives abound. Thanks Popper et al.

Types of Nulls

At least in the scope of Causal Inference. You a Sharp Null where you say “The Treatment had zero effect on all people”:

$$H_0: Y_i^1 - Y_i^0 = 0 \quad \forall i$$

Pretty bold! The next one is the Average Null where you’re saying the “The Treatment had zero effect on average”. I.e., the ACE is zero.

$$H_0: ACE = E[Y^1 - Y^0] = 0$$

These are not the same! A Sharp Null can imply an Average Null but the reverse is not true! Things can cancel out. Individual effects can be notable even when the overall average effect is zero.

Mister Fisher

The Gist

You treat the Potential Outcomes $Y^a$ as fixed. The Randomness comes from the Assignment $A$.

“Assign however your design says². Under my Null, nobody’s outcome changes no matter which assignment comes up. So I’ll ‘replay’ every assignment your design could have produced and see where yours lands.”

We cannot observe counterfactuals. That information is lost to the Ether. But if you assume that the Sharp Null is zero, you can ‘fill in’ the counterfactuals column! How does this help?

Consider the effect of some intervention on some outcome. Doesn’t matter what it is. You have two groups: Treatment and Control and you do a Complete Randomization and assign exactly two people to each group. You record a numerical value for each participant.

Student	$A$	Score
Alice	1	8
Ben	0	6
Carol	1	4
David	0	2

Okay now what?

• You say “I am going to use the Mean Score to examine the effect between the treatment and control groups.” You call this the Test Statistic $T_{\text{Observed}}$. From that table it is $\frac{(8 + 4)}{2} - \frac{(6 + 2)}{2} = +2$
• But you do not know what would have happened to Ben and David if they were assigned treatment. Assume the Sharp Null and you can draw this table. Under $Y_i^1 = Y_i^0$, the unobserved Potential Outcome is just the observed one so the bolded numbers are the ones you just filled in.

Student	$A$	$Y^0$	$Y^1$
Alice	1	8	8
Ben	0	6	6
Carol	1	4	4
David	0	2	2

Now what? Since it is a Complete Randomization, there are ${4 \choose 2} = 6$ possible assignments you could’ve chosen from. You picked one and the other 5 are lost. But assuming the Sharp Null, you can ‘fill in’ all five other assignments and see what it does to your Test Statistic!

$A_i$	Treatment	Control	$\text{Mean}_{\text{Treatment}}$	$\text{Mean}_{\text{Control}}$	Test Statistic
1	Alice, Ben	Carol, David	7	3	+4
2*	Alice, Carol	Ben, David	6	4	+2
3	Alice, David	Ben, Carol	5	5	0
4	Ben, Carol	Alice, David	5	5	0
5	Ben, David	Alice, Carol	4	6	−2
6	Carol, David	Alice, Ben	3	7	−4

Now what? You see that $T_i \in \{+4,+2,0,0,-2,-4 \}$. Each assignment has a probability $\frac{1}{6}$. This is the Randomization Distribution under the Null. You can now ask questions of it.

So far we have $T_{\text{Observed}} = +2$. We ask “What is the probability of seeing $T_i$ at least as extreme as our $T_\text{Observed}$?”

When we ask this Two-Sided-ly we are dealing with $|T_i| \ge |T_{\text{Observed}}|$. So here, we count $\{+4,+2,-2,-4\}$ and get $\frac{4}{6} = \boxed{\approx 0.67}$ or 67%.

Now what? What are we saying with 67%? Hark!

Even if the treatment did nothing, Random Assignment was 67% responsible for the $T_\text{Observed} = +2$ effect that we saw.

But that statement is not correct. Only a small step. We should qualify:

Under the Sharp Null (the treatment had no effect), 67% of all Random Assignments would produce an effect (difference in means) at least as big (in absolute value) as what we saw, $T_\text{Observed} = +2$

This is about how often $T_\text{Observed}$ would happen under the Null.

Another way to think is “How surprised are we at seeing what we saw if the Null were true?” 67% is not that surprising. So we fail to reject the Null. What’s the cutoff/threshold where reject it? Depends but a 5% value is typical.

This does not mean that our treatment did or didn’t work. We just don’t have enough evidence against the Null and should maybe try later. Maybe with a different design or with more people.

Why is this nice?

Notice that the distribution just ‘pops out’ of how we did things. You don’t need a Normal or other Distribution. This is called Design-Based Inference for this reason: the design of the experiement gives you the probability distribution that you can use for inference.

Will it Scale Bro?

We only had 4 people. What if we had 1,000 and assigned 500 to each group? In Complete Randomization that is ${1,000 \choose 500} \approx 2.7 \times 10^{299}$ assignments. See how shit blows up quickly? Can you do what we did before? Should you?

Nope. And even if you could, you can gather enough ‘signal’ from a small number of randomized assignments and don’t have to boil oceans. This is the Monte-Carlo Approximation of the Randomization Distribution. You simulate a small number of assignments and see if there’s anything happening.

Let’s say 10,000 random assignments. You find that 420 of these produce an effect (difference in means) at least as big (in absolute value) as what we saw $T_{\text{Observed}} = +2$.

$$\frac{420}{10{,}000} = 0.042$$

So the p-value here is 4.2%. This is approximate since we couldn’t be arsed to go through all 1,000 assignments. This is the probability if seeing what you saw ‘holding’/assuming the Null to be true (there is no effect.) At the typical cutoff of 5%, you reject the Null.

Moving on.

Mister Neyman

Here we focus on the average causal effect size. Fisher was a smoke test: “Is anything there?” This is a “How much is there?”

The Null Hypothesis here is “The average effect is zero.” Let’s use SACE. The null here is:

$$\text{SACE} = \frac{1}{N} \sum_{i=1}^N(Y_i^1 - Y_i^0) = 0$$

Remember that this is a God Mode claim. We cannot observe both potential outcomes (the Fundamental Problem of Causal Inference) and have to deal with that. We do this by putting a hat on SACE, the estimand, and estimating it as $\widehat{SACE}$, the estimator. A number will pop out. It is the estimate.

Again, this is the same setup as before. You are answering two different questions.

Say you have 20 people

$A$	Alice	Ben	Carol	David	Treated mean	Control mean	$\widehat{SACE}$
1	Caffeine (8)	Caffeine (7)	Nap (5)	Nap (3)	7.5	4.0	+3.5
2	Caffeine (8)	Nap (6)	Caffeine (5)	Nap (3)	6.5	4.5	+2.0
3	Caffeine (8)	Nap (6)	Nap (5)	Caffeine (4)	6.0	5.5	+0.5
4	Nap (6)	Caffeine (7)	Caffeine (5)	Nap (3)	6.0	4.5	+1.5
5	Nap (6)	Caffeine (7)	Nap (5)	Caffeine (4)	5.5	5.5	0.0
6	Nap (6)	Nap (6)	Caffeine (5)	Caffeine (4)	4.5	6.0	−1.5
Avg							+1.0

Footnotes

1. Using our estimator to estimate our estimand… ↩

2. This assignment is still randomized! We are not saying assign willy-nilly! Fisher cares a lot about Randomization. Which one you use will change the eventual Randomization Distribution and the p-value. With Complete Randomization here, we have ${4 \choose 2} = 6$ possible assignments. If you went Bernoulli, you’d have $2^4 = 16$ possible assignments. The distro and the p-value and inference would change under these two designs! ↩