ACTG · Computational approaches to brain health in HIV
From cohort data to cognitive insight
Raha Dastgheyb, PhD
Department of Neurology, Johns Hopkins University School of Medicine
Can we trust what goes into our analyses?
Codebooks · quality checks · SciDataReportR · QuartoWhat does a cognitive score actually mean?
BRACE · processing · normative modelsWhat patterns can we discover?
Exploration · unsupervised learning · clusteringCan we predict those patterns reliably?
Regression · random forests · validation · metricsWhat drives predictions, and what can we learn?
Variable importance · SHAP · DALEXThe purpose of computing is insight, not numbers.
Richard Hamming
Numerical Methods for Scientists and Engineers, 1962
01
Preparing the data
Can we trust what goes into our analyses?
What makes them valuable Decades of repeated visits · cognitive testing, clinical data, biospecimens · far cleaner than routine health-record data
What makes them hard
| Merges | A join silently drops 40 participants or duplicates 12 visits. |
|---|---|
| Missing data | −9, −99999, and blanks may mean different things. Not collected, skipped, and refused are not equivalent. |
| Variable types | A binary variable is treated as continuous. |
| Labels & definitions | Cryptic export names appear in figures, or collaborators use different labels and definitions. |
| Statistical choices | The statistical test doesn’t match the data or question. |
| Analysis history | A result depends on filters or recoding decisions that were never documented. |
None of these necessarily make the software throw an error. That’s what makes them dangerous.
To err is human; to really foul things up requires a computer.
Attributed to Paul R. Ehrlich
Every transformation.
Every exclusion.
Every decision.
Someone else should be able to see what I did and reproduce it.
Visible, traceable, correctable.
What changed?
The recode, calculation, filter or model choice.
Why did it change?
The rule, rationale and relevant assumptions.
What did it affect?
The participants, measurements and results.
Replication is the ultimate standard by which scientific claims are judged.
Roger D. Peng
Reproducible Research in Computational Science, 2011
Scan for the package and install instructions
Make it so.
Patrick Stewart as Jean-Luc Picard
Star Trek: The Next Generation
Why a package
Shared analysis and reporting foundation
In development
In development
In development
SciDataAgent? Concept / in development
Separating the code from the metadata
| Variable1,2 | Label | Type | Ordered Factor | n_missing | complete_rate | n_unique | Top Counts | Mean | Median | SD | Min | Max | Histogram |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HIVSTAT | HIV serostatus | factor | FALSE | 0 | 1 | 2 | PWH: 1874, PWO: 1063 | ||||||
| AGEATVIS | Age (years) | numeric | 0 | 1 | 59 | 55.21 | 56 | 11.34 | 28 | 87 | ▂▅▇▅▁ | ||
| BSEDUCTN | Education | factor | FALSE | 10 | 1 | 5 | Som: 942, Com: 629, Mor: 564, Les: 470 | ||||||
| SEXBIRTH | Sex at birth | factor | FALSE | 0 | 1 | 2 | Fem: 1552, Mal: 1385 | ||||||
| EducationYears | Education (estimated years) | numeric | 175 | 0.94 | 14 | 13.48 | 13 | 2.53 | 3 | 18 | ▁▁▇▆▆ | ||
| PHQ2 | PHQ-2 depressive symptoms (0-6) | numeric | 31 | 0.99 | 8 | 1.22 | 0 | 1.59 | 0 | 6 | ▇▂▁▁▁ | ||
| 1 Data Frame has 2937 rows and 6 columns | |||||||||||||
| 2 Color indicates type of variable | |||||||||||||
Failures in communication … resulted in a decision … based on incomplete and sometimes misleading information.
Presidential Commission on the Space Shuttle Challenger Accident
1986, Chapter V · Excerpt
There is nothing better than a picture for making you think of questions you had forgotten to ask.
John W. Tukey & Paul A. Tukey
quoted in Friedman & Stuetzle, Annals of Statistics, 2002
The greatest value of a picture is when it forces us to notice what we never expected to see.
John W. Tukey
Exploratory Data Analysis, 1977
| Measure | Mean | Maximum |
|---|---|---|
| Age as exported | 103.1 | 999 |
| Age after codebook recoding | 72.5 | 115 |
The code required to produce your output is part of the document itself.
Quarto Documentation
Tutorial: Computations
02
Measuring cognition
What does a cognitive score actually mean?
BRACE in the MWCCS: Trail Making A and B, Stroop-Color, visuospatial learning. Animations: brainbaseline.com
One file per task, for the whole study
| Task phase | Trial | Stimulus colour | Response colour | Correct response | Response time (ms) |
|---|---|---|---|---|---|
| Practice | 1 | Red | Blue | No | 940 |
| Practice | 2 | Blue | Blue | Yes | 820 |
| Test | 1 | Green | Green | Yes | 760 |
| Test | 2 | Red | Red | Yes | 705 |
| Test | 3 | Blue | Green | No | 650 |
| Test | 4 | Green | Green | Yes | 810 |
| Test | 5 | Red | Red | Yes | 735 |
| Test | 6 | Blue | Blue | Yes | 790 |
There’s so much data out there, but there’s a finite amount of truth.
Nate Silver
WIRED, 2012
github.com/BHPDataSci/BraceExtraction
32 years old, college graduate Trail Making A: 22 seconds
T = 36
68 years old, did not finish high school Trail Making A: 22 seconds
T = 52
Illustrative people, scored with the published model.
The normative idea
A primary goal of clinical neuropsychological assessment is to determine whether a given set of test results suggests brain pathology.
Robert K. Heaton and colleagues
Demographic Influences on Neuropsychological Test Performance, 1996
norm_TrailsA <- CreateNormativeTScoreModel(
data = df_BRACE,
test_var = "Trails_TrailsA_totalTime", # the raw score: completion time in milliseconds
covariates = c("AGEATVIS", "Education"), # what we adjust for
reference_var = "HIVStatus", # who defines "expected":
reference_value = "PWOH", # people without HIV
convert_seconds = TRUE, # milliseconds -> seconds
log_transform = TRUE, # timed scores have a long tail of slow runs
reverse_score = TRUE # flip the sign, so higher always means better
)norm_TrailsA$plots$raw
norm_TrailsA$plots$scaled
norm_TrailsA$plots$tscore
The saved model: coefficients from people without HIV
| Term | Estimate | SE |
|---|---|---|
| (Intercept) | -1.0661 | 0.0296 |
| AGEATVIS | -0.0041 | 0.0005 |
| Education< High school | -0.0256 | 0.0201 |
| EducationSome college | 0.0654 | 0.0170 |
| EducationCollege | 0.1264 | 0.0207 |
| Education> College | 0.1257 | 0.0181 |
Residual SD = 0.187. The outcome is −log10(seconds).
The same steps, applied to two people who both took 22 seconds
| Person | 1 · Raw (s) | 2 · −log10 | 3 · Expected | 4 · Difference ÷ SD | 5 · T = 50 + 10z |
|---|---|---|---|---|---|
| 32 y, college graduate | 22 | -1.342 | -1.072 | -1.45 | 36 |
| 68 y, < high school | 22 | -1.342 | -1.372 | 0.16 | 52 |
The model is frozen The equation and every preprocessing step are stored in one object. The same object scores people with HIV, a later visit, another site or another study, so T-scores are comparable across papers.
Trail Making A. Grey line: no demographic adjustment. Orange: the norm you selected.
The most dangerous phrase in the language is, ‘We’ve always done it this way.’
Attributed to Grace Hopper
2,937
participants, each assessed once, Nov 2020 to Mar 2025
1,063
people without HIV: the reference group
| Characteristic | PWOH N = 1,0631 |
PWH N = 1,8741 |
p-value2 | Test2 | Effect size | ES method |
|---|---|---|---|---|---|---|
| Age (years) | 56.95 (12.34) | 54.22 (10.61) | <0.001 | Welch t-test | 0.24 | |d| |
| Education | <0.001 | Pearson chi-squared | 0.12 | Cramer's V | ||
| High school | 201 (19%) | 428 (23%) | ||||
| < High school | 152 (14%) | 318 (17%) | ||||
| Some college | 307 (29%) | 635 (34%) | ||||
| College | 142 (13%) | 180 (9.6%) | ||||
| > College | 258 (24%) | 306 (16%) | ||||
| Sex at birth | <0.001 | Pearson chi-squared | 0.09 | Cramer's V | ||
| Female | 499 (47%) | 1,053 (56%) | ||||
| Male | 564 (53%) | 821 (44%) | ||||
| Trail Making A T-score | 50.00 (9.98) | 49.98 (9.83) | 0.954 | Welch t-test | 0.00 | |d| |
| Trail Making B T-score | 50.00 (9.98) | 49.52 (10.06) | 0.213 | Welch t-test | 0.05 | |d| |
| Stroop-Color response time T-score | 50.00 (9.98) | 49.08 (10.14) | 0.0177 | Welch t-test | 0.09 | |d| |
| Stroop-Color accuracy T-score | 50.00 (9.98) | 48.96 (10.86) | 0.00862 | Welch t-test | 0.10 | |d| |
| Visuospatial learning (4x4) T-score | 50.00 (9.98) | 49.43 (9.88) | 0.137 | Welch t-test | 0.06 | |d| |
| 1 Mean (SD); n (%) | ||||||
| 2 p-values use Welch t-test (two groups) or ANOVA (more than two groups) for continuous outcomes and chi-square or Fisher’s exact test, selected from expected counts for categorical outcomes. | ||||||
03
Learning from the data
What patterns can we discover?
See, and then reason and compare and control. But see first.
William Osler
I do not think it means what you think it means.
Inigo Montoya
The Princess Bride
What do we mean by impairment?
Why look at profiles?
Nightingale S et al. (2023). Cognitive impairment in people living with HIV: consensus recommendations for a new approach. Nature Reviews Neurology 19:424–433. · Mukerji SS et al. (2023). Machine learning approaches to understand cognitive phenotypes in people with HIV. Journal of Infectious Diseases 227:S48–S57. · Dastgheyb RM et al. (2021). Patterns and predictors of cognitive function among virally suppressed women with HIV. Frontiers in Neurology 12:604984.
Choose the task before choosing the model.
Cluster the map, not the people
Less noise. Each node is a local average of similar people.
Cheaper. Cluster a hundred nodes instead of thousands of people.
Shape, not just level. Clustering raw test scores tends to find “high, medium, low”, which is the global score again.
The map compresses people into nodes laid out on a grid, with similar nodes next to each other.
Mclust groups the nodes into k clusters and gives every node a probability of belonging to each.
Map back gives each participant the cluster of her closest node.
Left: the data. Right: the same nodes on the grid. Toy data.
Grey: people · orange: nodes · lines join grid neighbours
Early: a wide neighbourhood moves the whole net. Late: only the winner moves.
929
virally suppressed women in the WIHS
17
test T-scores, not domain scores
The design
One call does all of it Scale the scores · train the map · cluster the nodes with every combination of k and model family · compare the candidates · map back to people.
Exploratory first Exploratory mode compares solutions. Once one is chosen, the same function refits it in "finalize" mode to give a model that can be applied to new people.
04
Building and validating models
Can we predict those patterns reliably?
| Characteristic | Unimpaired N = 3111 |
Executive function (sequencing) N = 1291 |
Speed N = 1441 |
Learning N = 1371 |
Learning & memory N = 861 |
Global weakness (processing speed) N = 1221 |
p-value2 | Test2 | Unimpaired - Executive function (sequencing) | Unimpaired - Speed | Unimpaired - Learning | Unimpaired - Learning & memory | Unimpaired - Global weakness (processing speed) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Age | 46.70 (8.07) | 45.84 (8.65) | 47.34 (8.94) | 46.78 (8.46) | 47.36 (8.08) | 44.65 (9.26) | 0.104 | ANOVA | 1 | 1 | 1 | 1 | 0.165 |
| Education | 12.80 (3.26) | 13.02 (2.90) | 12.47 (3.02) | 12.39 (2.69) | 12.13 (2.88) | 12.87 (2.91) | 0.192 | ANOVA | 1 | 1 | 0.83 | 0.316 | 1 |
| Race | 0.0451 | Fisher (sim.) | 0.0185 | 0.384 | 1 | 0.62 | 0.301 | ||||||
| black non-hispanic | 202 (65%) | 103 (80%) | 104 (72%) | 94 (69%) | 65 (76%) | 89 (73%) | |||||||
| hispanic | 45 (14%) | 7 (5.4%) | 15 (10%) | 18 (13%) | 12 (14%) | 16 (13%) | |||||||
| other | 8 (2.6%) | 5 (3.9%) | 8 (5.6%) | 3 (2.2%) | 2 (2.3%) | 6 (4.9%) | |||||||
| white | 56 (18%) | 14 (11%) | 17 (12%) | 22 (16%) | 7 (8.1%) | 11 (9.0%) | |||||||
| Income | <0.001 | Pearson chi-squared | 0.726 | 0.0136 | 0.859 | <0.001 | 0.121 | ||||||
| Less than 12k/y | 126 (41%) | 62 (48%) | 80 (56%) | 65 (47%) | 60 (70%) | 64 (52%) | |||||||
| More than 12ky | 185 (59%) | 67 (52%) | 64 (44%) | 72 (53%) | 26 (30%) | 58 (48%) | |||||||
| Employed | 152 (49%) | 47 (36%) | 41 (28%) | 58 (42%) | 23 (27%) | 43 (35%) | <0.001 | Pearson chi-squared | 0.085 | <0.001 | 1 | 0.00127 | 0.0517 |
| Depressive Symptoms | <0.001 | Pearson chi-squared | 0.218 | 0.00636 | 0.027 | <0.001 | 0.0124 | ||||||
| Currently Depressed | 66 (21%) | 39 (30%) | 51 (35%) | 46 (34%) | 39 (45%) | 43 (35%) | |||||||
| Not Depressed | 245 (79%) | 90 (70%) | 93 (65%) | 91 (66%) | 47 (55%) | 79 (65%) | |||||||
| Perceived Stress | 0.00113 | Pearson chi-squared | 0.0151 | 0.286 | 0.934 | <0.001 | 0.127 | ||||||
| Elevated Percieved Stress | 230 (74%) | 77 (60%) | 94 (65%) | 93 (68%) | 44 (51%) | 77 (63%) | |||||||
| Not Elevated Percieved Stress | 81 (26%) | 52 (40%) | 50 (35%) | 44 (32%) | 42 (49%) | 45 (37%) | |||||||
| Post-traumatic stress | 39 (13%) | 32 (25%) | 24 (17%) | 27 (20%) | 26 (31%) | 19 (16%) | <0.001 | Pearson chi-squared | 0.00765 | 1 | 0.215 | <0.001 | 1 |
| Crack, cocaine, heroin | <0.001 | Fisher (sim.) | 0.34 | 1 | 1 | 1 | <0.001 | ||||||
| Current User | 16 (5.1%) | 4 (3.1%) | 9 (6.3%) | 8 (5.8%) | 6 (7.0%) | 5 (4.1%) | |||||||
| Former User | 170 (55%) | 58 (45%) | 79 (55%) | 66 (48%) | 46 (53%) | 35 (29%) | |||||||
| Never Used | 125 (40%) | 67 (52%) | 56 (39%) | 63 (46%) | 34 (40%) | 82 (67%) | |||||||
| Marijuana Use | 0.00211 | Pearson chi-squared | 1 | 1 | 1 | 1 | <0.001 | ||||||
| Former | 170 (55%) | 63 (49%) | 77 (53%) | 68 (50%) | 49 (57%) | 54 (44%) | |||||||
| Never | 78 (25%) | 43 (33%) | 45 (31%) | 44 (32%) | 24 (28%) | 59 (48%) | |||||||
| Recent | 63 (20%) | 23 (18%) | 22 (15%) | 25 (18%) | 13 (15%) | 9 (7.4%) | |||||||
| Nadir CD4 | 305.68 (235.61) | 309.68 (237.58) | 306.36 (229.22) | 310.75 (217.87) | 321.80 (214.21) | 303.67 (205.78) | 0.994 | ANOVA | 1 | 1 | 1 | 1 | 1 |
| Current CD4 | 677.05 (343.19) | 660.29 (316.82) | 704.62 (312.29) | 681.88 (310.92) | 629.50 (286.48) | 664.42 (288.29) | 0.627 | ANOVA | 1 | 1 | 1 | 0.973 | 1 |
| Years of effective ART | 7.93 (4.75) | 7.59 (4.59) | 8.33 (4.29) | 7.77 (4.59) | 8.02 (4.49) | 8.47 (4.45) | 0.619 | ANOVA | 1 | 1 | 1 | 1 | 1 |
| 1 Mean (SD); n (%) | |||||||||||||
| 2 p-values use Welch t-test (two groups) or ANOVA (more than two groups) for continuous outcomes and chi-square or Fisher’s exact test, selected from expected counts for categorical outcomes. Pairwise p-values are bonferroni-adjusted within outcome. | |||||||||||||
Nowhere is it written on a stone tablet what kind of model should be used to solve problems involving data.
Leo Breiman
Statistical Modeling: The Two Cultures, 2001
Macchine learning models can pick up on nonlinear patterns
Trust, but verify.
Ronald Reagan
Quoting a Russian proverb, 1987
What will generalize?
Model selection, internal validation and testing in a new population answer different questions.
One partition to develop the model. One untouched partition for its final evaluation.
Each tile is one person. Splits must respect repeated visits, families or sites when relevant.
Sample people with replacement. Fit on the resample and assess optimism or use observations left out of that draw.
Numbers show how many times each person was drawn. Zero means left out. Use a defined bootstrap validation estimator.
Outer resampling estimates performance of the complete selection process.
Inner resampling selects model settings using only the current outer training sample.
Outer training participants
Inner folds select the model settings
Refit on the outer training participants
Assess once on the outer assessment participants
Repeat across outer folds. Keep every model-selection decision inside the outer training sample.
Development cohort
Fit and select the workflow.
Freeze the scoring rules, transformations, model and decision threshold.
New setting
Evaluate in later participants, another site or an independent cohort.
Check calibration, discrimination and clinically relevant errors.
Internal resampling cannot establish transportability to a population it never observed.
Does that encapsulate you?
Robin Williams as Sean Maguire
Good Will Hunting, 1997
| Observed | ||
|---|---|---|
| Predicted | In the profile | Unimpaired |
| In the profile |
TP True positive |
FP False positive |
| Unimpaired |
FN False negative |
TN True negative |
The clinical question determines which errors matter. Performance also depends on the threshold and the population.
05
Explaining our models
What drives predictions, and what can we learn?
2026 reconstruction
One example: the processing-speed profile versus the unimpaired group. Each point is an original participant.
Left: contribution away from the profile. Right: toward it. Categorical colors identify levels, without implying a biological order.
Across the cohort: which pattern did the model learn?
Centered response, in percentage points (ALE)
For one woman: how does her prediction change?
Predicted probability, with one example highlighted (ICE)
Left: a centered summary across the original cohort. Right: individual predictions with other inputs fixed. These describe the model, not treatment effects.
Clusters are a description The algorithm will always return groups. Ask whether they are stable, and whether they predict anything you did not cluster on.
Performance before interpretation Importance and SHAP can be unstable when prediction is weak. Report validation and check rank stability.
Prediction is not causation A variable can be important because it stands in for something unmeasured.
Every upstream choice is in the result The norm, the quality rules, the tests chosen. Change them and the clusters can change.
New people are the real test Freeze the model and apply it to another cohort or another study.
Write it down Code, codebook, and versions, so someone else can rerun it and disagree with you.
1
The pipeline is part of the science. Merges, codes, quality rules and norms are scientific decisions. Make them visible and rerunnable.
2
A score is an interpretation. The reference group and the covariates decide who falls below the line.
3
Every summary compresses. Before collapsing to one label, ask whether the answer was in the variation.
4
Machine Learning Models are Hypothesis-generating. Not diagnostic.
R01MH125300 · R21MH125278 · MH113512 · NSF19-500 · MH119947 · MH120693 · P30MH075673 · P30AI094189 · R03MH123290 (Dastgheyb) · CFAR Scholar Award (Dastgheyb) · Michael and Ann Hankin Fund (Dastgheyb)