Power and Sampling

ANOVA tells you difference between groups but doesn’t tell you what the difference is. E.g. you have a control arm and two intervention arms.

TODO: What’s the point of the Bonferroni correction? YOu need to adjust for multiple comparisons.

Now science and statistics evolved independently (sort of). It should be okay to enumerate all hypotheses and not just the one that worked out of ‘shame’. There’s a move to reporting CI’s instead of p-values.

Sample size is affected by

1. Significance Level $\alpha$ (lower means more)
2. Whether it’s a one or two tailed test (latter needs more) - Signal and directionality
3. Effect Size (smaller means more)
4. Power ($1 - \beta$) (larger means more) -

Power

Detect differences or relationshipts that actually exist in the population. You are looking for robust phenomena that you can replicate. This helps with the reliability and validity of your study and science itself. Think of this as the ‘strength of the signal’ of relationships or differences or something of interest in the world.

What are the problems with small and large sample sizes? TODO

Small sample size is the most common reason for Type II error ($\beta$).

Parametric tests give you more power. $\chi^2$ is the weakest of the statistical tests.

More sample size doesn’t necessarily mean better power.

Analysis

You have $\alpha$, sample size $n$, effect size, and power ($1 - \beta$). If you know three you determine the fourth. To select sample size:

1. State Null
2. 1/2 tailed?
3. Select test
4. Select ES
5. Select $\alpha$ and $\beta$

Then look up some tables. Comparing between groups will be different test.

Effect Size

What’s present in the population is the Effect. The extent of it’s presence is the Effect Size. It’s dimensionless. Now the number is $[0,1] \in R^+$ and typically 0.5 is “large” and 0.3-0.5 is “medium”. Your best bet is to look in literature for metaanalyses.

$$\text{Standardized Effect Size} = \frac{\bar{X}_{\text{Treatment}} - \bar{X}_{\text{control}}}{SD_{\text{Control or Treatment}}}$$

In a regression, the ES is the smallest correlation Coefficient we’d like.

Examples

See Hulley 2007 Example 1 from lectures. Note that there is the Effect Size of $10\% \times 2.0L$ and the Standardized Effect Size where you divide by the SD. It’s a two-sided test (directionality doesn’t matter). Null is that there is no difference in effects of the two drugs. See Hulley 2007 Example 2. The ES is already given in the question but $P_2 - P_1$ is $(0.3 - 0.2) = 0.1$

See Hulley 2007 Example 3.

Chi-squared Test

Always two-sided. For categorical variables. $ES = P_1 - P_2$ where $P$ is the proportion.