TLDR: Fisher Tests, Neyman Estimates
| Step | Fisher | Neyman |
|---|---|---|
| Start with | No effect. All Lies! | ACE (the Estimand (God Mode!)) |
| Why Rando? | Generate distribution under the sharp null | For Unbiasedness and to calculate uncertainty |
| Calculation | Compare against differences from possible assignments | Estimate the effect as and calculate its standard error |
| Result | p-value | An estimate and confidence interval |
| Conclusion | Evidence for or against the sharp null | Magnitude and uncertainty of ACE |
Pre-Flight
So we've randomized. We do the experiment. We may see something interesting. How do we know this is not due to pure chance? Due to some other funky stuff going on?
Mister Fisher
Mister Fisher
Mister Neyman
Here we focus on the average causal effect size. Fisher was a smoke test: "Is anything there?" This is a "How much is there?"