Fisher

Mister Fisher

The Gist

You treat the Potential Outcomes $Y^a$ as fixed. The Randomness comes from the Assignment $A$.

“Assign however your design says¹. Under my Null, nobody’s outcome changes no matter which assignment comes up. So I’ll ‘replay’ every assignment your design could have produced and see where yours lands.”

We cannot observe counterfactuals. That information is lost to the Ether. But if you assume that the Sharp Null is zero, you can ‘fill in’ the counterfactuals column! How does this help?

Consider the effect of some intervention on some outcome. Doesn’t matter what it is. You have two groups: Treatment and Control and you do a Complete Randomization and assign exactly two people to each group. You record a numerical value for each participant.

Student	$A$	Score
Alice	1	8
Ben	0	6
Carol	1	4
David	0	2

Okay now what?

• You say “I am going to use the Mean Score to examine the effect between the treatment and control groups.” You call this the Test Statistic $T_{\text{Observed}}$. From that table it is $\frac{(8 + 4)}{2} - \frac{(6 + 2)}{2} = +2$
• But you do not know what would have happened to Ben and David if they were assigned treatment. Assume the Sharp Null and you can draw this table. Under $Y_i^1 = Y_i^0$, the unobserved Potential Outcome is just the observed one so the bolded numbers are the ones you just filled in.

Student	$A$	$Y^0$	$Y^1$
Alice	1	8	8
Ben	0	6	6
Carol	1	4	4
David	0	2	2

Now what? Since it is a Complete Randomization, there are ${4 \choose 2} = 6$ possible assignments you could’ve chosen from. You picked one and the other 5 are lost. But assuming the Sharp Null, you can ‘fill in’ all five other assignments and see what it does to your Test Statistic!

$A_i$	Treatment	Control	$\text{Mean}_{\text{Treatment}}$	$\text{Mean}_{\text{Control}}$	Test Statistic
1	Alice, Ben	Carol, David	7	3	+4
2*	Alice, Carol	Ben, David	6	4	+2
3	Alice, David	Ben, Carol	5	5	0
4	Ben, Carol	Alice, David	5	5	0
5	Ben, David	Alice, Carol	4	6	−2
6	Carol, David	Alice, Ben	3	7	−4

Now what? You see that $T_i \in \{+4,+2,0,0,-2,-4 \}$. Each assignment has a probability $\frac{1}{6}$. This is the Randomization Distribution under the Null. You can now ask questions of it.

So far we have $T_{\text{Observed}} = +2$. We ask “What is the probability of seeing $T_i$ at least as extreme as our $T_\text{Observed}$?”

When we ask this Two-Sided-ly we are dealing with $|T_i| \ge |T_{\text{Observed}}|$. So here, we count $\{+4,+2,-2,-4\}$ and get $\frac{4}{6} = \boxed{\approx 0.67}$ or 67%.

Now what? What are we saying with 67%? Hark!

Even if the treatment did nothing, Random Assignment was 67% responsible for the $T_\text{Observed} = +2$ effect that we saw.

But that statement is not correct. We should qualify:

Under the Sharp Null (the treatment had no effect), 67% of all Random Assignments would produce an effect (difference in means) at least as big (in absolute value) as what we saw, $T_\text{Observed} = +2$

This is about how often $T_\text{Observed}$ would happen under the Null.

You are friends with the Null, like a good scientist. You want to help the Null. The Null says “foobar”. You say “You’re right! I will give you evidence that in 67% of all random assignments, foobar!”. So you fail to reject the Null. Conversely, you and Null will need a small chit-chat if you say “Ummm. I can give you evidence that in only 1% of all random assignments will foobar.” The Null is unhappy but you break up first by rejecting the Null.

A 5% cutoff is typical.

This does not mean that our treatment did or didn’t work. We just don’t have enough evidence against the Null and should maybe try later. Maybe with a different design or with more people.

Why is this nice?

Notice that the distribution just ‘pops out’ of how we did things. You don’t need a Normal or other Distribution. This is called Design-Based Inference for this reason: the design of the experiement gives you the probability distribution that you can use for inference.

Will it Scale Bro?

We only had 4 people. What if we had 1,000 and assigned 500 to each group? In Complete Randomization that is ${1,000 \choose 500} \approx 2.7 \times 10^{299}$ assignments. See how shit blows up quickly? Can you do what we did before? Should you?

Nope. And even if you could, you can gather enough ‘signal’ from a small number of randomized assignments and don’t have to boil oceans. This is the Monte-Carlo Approximation of the Randomization Distribution. You simulate a small number of assignments and see if there’s anything happening.

Let’s say 10,000 random assignments. You find that 420 of these produce an effect (difference in means) at least as big (in absolute value) as what we saw $T_{\text{Observed}} = +2$.

$$\frac{420}{10{,}000} = 0.042$$

So the p-value here is 4.2%. This is approximate since we couldn’t be arsed to go through all 1,000 assignments. This is the probability if seeing what you saw ‘holding’/assuming the Null to be true (there is no effect.) At the typical cutoff of 5%, you reject the Null.

Moving on.

Footnotes

1. This assignment is still randomized! We are not saying assign willy-nilly! Fisher cares a lot about Randomization. Which one you use will change the eventual Randomization Distribution and the p-value. With Complete Randomization here, we have ${4 \choose 2} = 6$ possible assignments. If you went Bernoulli, you’d have $2^4 = 16$ possible assignments. The distro and the p-value and inference would change under these two designs! ↩