Pre-Flight

So we’ve randomized. We do the experiment. We may see something interesting¹. How do we know this is not due to pure chance? Due to some other funky stuff going on?

Fisher tests, Neyman estimates

The 10,000ft overview is:

Question	What	Who
Did it do anything?	Hypothesis Testing	Fisher
How much did it do?	Effect Estimation	Neyman

Few more sentences:

• Fisher: I think the treatment does nothing to anyone. FIGHT ME.
• Neyman: I just wanna know how much does the treatment change the outcome on average and the precision with which you can say this. I think it’s zero 💅

A Reminder on The Null

The Null hypothesis for non-sophisticated people is: Nothing Happened. There was no effect. The average mean was zero. The ratio was one. And so on. The World is as it is before and after your intervention. You start with this, a most conservative position.

You also never ‘prove’ the Null or your Alternate hypothesis. You either reject or fail to reject the Null. Double-negatives abound. Thanks Popper et al.

Types of Nulls

At least in the scope of Causal Inference. You a Sharp Null where you say “The Treatment had zero effect on all people”:

$$H_0: Y_i^1 - Y_i^0 = 0 \quad \forall i$$

Pretty bold! The next one is the Average Null where you’re saying the “The Treatment had zero effect on average”. I.e., the ACE is zero.

$$H_0: ACE = E[Y^1 - Y^0] = 0$$

These are not the same! A Sharp Null can imply an Average Null but the reverse is not true! Things can cancel out. Individual effects can be notable even when the overall average effect is zero. In other words: a zero average null does not mean that the treatement was ineffective!

Footnotes

1. Using our estimator to estimate our estimand… ↩