Day 02
There are all sorts of R packages that address each of the study questions below.
Disease Natural History
Characterization might seem ‘boring’ to some but it is a big chunk and important chunk of observational/clinical studies. Was discussed as “Table 1”. Just this step can be extraordinarily insightful on its own (never mind the fact that it’s upstream of the “shiny” thing you may want to study.)
It’s not ‘easy’ either. Think of COVID time. New infectious, disease. Affects special populations like pregnant women who are typically excluded from these types of studies (unless you’re explicitly studying them that is.)
Treatment Utilization
In one study, they found that ~10% of patients had a treatment pathway that was kinda bespoke. Now you need to think of what kinds of questions this prompts. E.g. What is the kind of patient that chooses a particular treatment (e.g. citalopram versus sertraline)?
Outcome Analysis
What happened as a result of the exposure? Look at incidence rates (Target population, Outcome, Time-at-risk TAR, Strata). The CohortIncidence package deals with this. \text{Incidence Proportion} = \frac{\text{People with Outcome during TAR}}{\text{# of people}} while .
See: OHDSI’s role in “turning on” the AstraZeneca vaccine for 700M people (they were worried about blood clots — “thromboemolic events”.)
Time to Event Specification
Combos are possible!
Who? When? Risk Factors…
Comparative Cohort Design (Study Design)
- Measure - Characterization
- Compare - Estimation
- Prediction
Active Comparator Design: Ethics (think of chemotherapy) is a big reason. So you compare ‘similar’ groups.
Association versus Causation. Confounders induce an association.
Using a Prop Score, you are trying to mimic all the blessings of an RCT (to minimize the effect of Confounders.)
TODO: See George’s diagram.
Manually picking confounders from literature is messy. There may be terrible overlap/agreement. So you can end up putting all of them into a bucket. Too many. Models suffer.
Another way is to have the data tell you what the confounders are (empirical selection, HDPS, high dim prop score.)
Yet another way is “Large-scale Propensity Score” (LSPS).
TODO: M-bias — correlation with underlying causes. Mediators, Colliders, Instruments (?)
Propensity score: you can say “this person is 1.5 people”. This is inverse probability weighting (TODO: overlap weighting). You reweight cohorts so the populations seem similar (TODO: explore more… check out the Garbe et al paper (2013))
Are you balancing covariates? There’s the Standardized Difference of Means . Graham says a diff of indicates negligible difference.
TODO: Equipoise…
TODO: Instruments?
TODO: “INducing bias”
Degrees of freedom to deal with non-linearity w.r.t. age (you bin them for this reason.)
You can never claim that you don’t have unmeasured confounders.
TODO: Active Comparators and Equipoise! Comparability is the key!
Start in your head with “If I could do a theoretical RCT, what would that look like?” Now, since you cannot randomize in observational studies, tweak it. Propensity scores. Active Comparators. Equipoise. Go bananas, make it good. E.g. How would you RCT answer “Does psoriasis cause GI bleeds?” You’re going to induce psoriasis in a group? Lol. Nothing wrong with doing association studies: you’re just not doing Causal Inference.
Patient-Level Prediction
Yeah ML is sexy but the problem specification is the most important thing.
We are not doing causal inference. What will happen to you is separate from why it happened.
Also: in ML and for prediction tasks, you cannot look into the future to predict the future. Watch out for leakage! Index time is everything!
A prediction problem has three elements: Observation Window (before and to ), Time-at-Risk, and the Outcome that happens within TAR.
How often does an adverse event happen as a result of a drug? Is it due to the drug? When you read the giant list of side effects, are they due to the drug?
Consider three entirely separate questions:
- Do people with ACEi get angioedemas?
- Does taking an ACEi cause angioedema?
- If I were to take an ACEi what is my risk of angioedema?
So you have a model. What do you do next? As a field we’re pretty bad at deploying this into the clinic (see this paper: very few are deployed! It’s a big ‘Last Mile’ problem!)
Now how do you specify the prediction task? Answer three questions:
- Whom do you want to do the prediction for?
- What do you want to predict?
- Over what time? (e.g. 30 days later? one year later? TODO: Is this always to be present?)
The PatientLevelPrediction library in OHDSI does this for you.
If your model fails (e.g. AUROC = 0.51), you have learned something! That something’s not predictable, at least in the way in which you’ve specified the task.
That’s Science™ too. Tell people about it. Save them time. And don’t make another sepsis prediction model, please.
Pedantry (justified): Don’t use “Risk Factors”, use “Predictors” instead.
In healthcare, your linear model is not textbook linear. Think about it.
Deployment Thought experiment: What would you do if I gave you an AUROC=0.9 model? Would deployment be justified (validating, calibrating, etc.)? This is why, again, the ML part is the easiest part in healthcare settings.
Tidbit: Death is arguably one of the most important but one of the least captured. You can try claims data to infer death (myocardial infraction → hospice → you vanish).