Skip to main content

Experimental Study Design

Experiment > Observation > Accepted Knowledge > Authority/Expertise

In an experiment, the investigator assigns the exposure/intervention. These are only prospective in nature (think about it).

RCT​

The bread-n-butter study design for hypothesis testing is the hallowed Randomized Controlled Trial (RCT). The idea is that you pick people (based on criteria) and randomly assign them to one of the intervention group or the control group. This allows you to minimize bias because the hope is that you’re distributing confounders equally across both groups1.

They’re expensive, slow, and can cause adverse effects.

Now with the pace of technology, you really want a way to collect results faster (think about how Chatbots have evolved… RL based chatbot is ‘beaten’ by ChatGPT).

There’s also an idea of the “Run-In Period”. You ‘warm up’ the study to see who will actually stick to the study.

There are a few elements:

  • Intervention (Intensity, Duration, Frequency) and Control definitions
  • Outcome measurements
  • Participants
  • Baseline Variables
  • Randomizing and Blinding (where you hide who’s exposed to what)
warning

No matter how many outcomes you’re interested in, in an RCT there is always one primary outcome and that’s what you’ll propose and report.

Outcomes​

Outcomes can be:

  • L1: Clinical Outcomes (Mortality, for example). These take a very long time.
  • L2: Intermediate markers
  • L3: Processes
  • Adverse and Unanticipated Effects (you kinda stop trials here)

Now in the example of diabetes, you can do some shortcutting: just say you want to lower A1C levels and people will just know that it has larger downstream effects on mortality. Lots of studies linking that biomarker to mortality. So you don’t have to say that you’re pursuing lower mortality as a goal. Quicker, simpler.

Participants​

You want to maximize the power of your study, minimize confounders, maximize benefit, and generalize broadly.

You need to balance between being too restrictive and too inclusive.

Use stratification to select for specific characteristics.

Baseline Variables​

You have to have an idea of of who’s in your sample. This is why you need to establish some baseline characteristics of the people and of the outcome that you wish to study in your RCT.

Randomizing​

Address bias here. You do this… randomly. There are several types starting with the simplest. But if you suspect that there are baselines that may affect your outcome (age, education, language (as a proxy for cultural differences esp)) you take these ‘chunks’ and randomly distribute from these chunks. This is not the same as block randomization! There’s also stratified randomization.

Now can you ‘chunk’ at the group level: think of deploying your intervention across two clinics. This is a Cluster Randomized Trial. You need more participants and may have decreased power.

Blinding​

Knowing that you’re in an intervention group and control group can create bias. You can go a step further and double-blind (experimenters and participants don’t know who is where).

Factorial Design​

So you want to study Drug A and Drug B (multiple interventions). Do a combinatorial thing:

  • Drug A + Placebo B
  • Placebo A + Drug B
  • Drug A + Drug B
  • Placebo A + Placebo B

So now you’ve kinda blown up your RCT to more than two arms!

Crossover Randomized Trial​

Two arms still. Now you perform intervention. Then you wait (“washout period”). Then you flip the treatment and control arms: you give the treatment to the control and vice-versa.

So now, you can compare between-group and within-group differences! Minimizes confounding! But takes forever.

Quasi-Experiments​

For when a ‘pure’ RCT is not possible: you need to find a way to have a comparison group Like if you wanted to test a CDS at an ICU. Take two ICUs one with CDS and another without.

Pre-Post Design with Concurrent Controls is the most commonly used experimental design for this purpose! Remember confounders? Here when you are not in God-mode, you try your best to understand the confounders on ‘either side’ and explain them and control for them (Residents are younger, shittier documentation practices).

Multiple measurements: You cannot measure stuff in July every year and claim improvements or declines (new residents!) You have to measure constantly. Now note that you can really irritate people by deploying the same questionnaire over and over! This can affect measurement!

Time-Series Design​

TODO

Footnotes​

  1. Theoretically if you have an infinite number of people per group you may have eliminated all confounder effects. ↩