Reproducibility Supplement

A Machine Learning Model That Incorporates CD45 Mean Flouoresence Intensity (MFI) Predicts Poor Viability of Hematopoietic Progentitor Cells After Freeze-Thaw

Authors
Affiliations
Arwa Z. Al-Riyami

Department of Hematology, Sultan Qaboos University Hospital

Elena Maryamchik

Department of Laboratory Medicine, Memorial Sloan Kettering Cancer Center

Richard S. Hanna

Department of Pediatrics, Children’s Hospital of Philadelphia

Amir Reza Pashmineh Azar

Blood and Bone marrow Transplant Program, Children’s Hospital of Philadelphia

Xingwu Zheng

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Shilpa Choudhari

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Colleen Finn

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Nicholas Giacobbe

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Rene Machietto

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Robert Rieser

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Farzaneh Ghasemi Tahrir

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Xiaoyong Zhang

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Stephan Kadauke

Department of Clinical Pathology & Laboratory Medicine, Perelman School of Medicine at the University of Pennsylvania

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Yongping Wang

Department of Clinical Pathology & Laboratory Medicine, Perelman School of Medicine at the University of Pennsylvania

Cell & Gene Therapy Laboratory, Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia

Published

November 15, 2022

Introduction

In this document, we provide the computer code that was used to create the tables, figures, and statistical test results of the primary analysis reported in our manuscript A Machine Learning Model That Incorporates CD45 Mean Flouoresence Intensity (MFI) Predicts Poor Viability of Hematopoietic Progentitor Cells After Freeze-Thaw (Al-Riyami et al). We provide this document to allow other interested parties to inspect, verify, and reproduce our analytic work.

The first part, Analyses, lists concise descriptions of the Analyses we performed. To view the code (written in the R programming language) that was used to perform a specific analysis, as well as the results, click the black triangle to the left of the description to expand each section. The order of this part follows the order of analyses reported in the text of the main manuscript. The tables and figures are cross-referenced.

The final part describes the precise Computational Environment in which the computational analyses were performed. Among other parameters, this part lists the names, versions, and sources of all software packages that were used to generate this document.

Analyses

Loading packages 
library(tidyverse)
library(lubridate)
library(readr)
library(tidymodels)
library(visdat)
library(here)
library(tidyr)
library(DT)
library(cgtr)
library(themis)
library(gtsummary)
library(flextable)
Loading data 
data_file <- here("qmd/cd34_model_data_20220915.csv")

cd34 <- read_csv(data_file) %>%
  mutate(across(where(is.character), as.factor)) %>% 
  mutate(poor_recovery = as_factor(case_when(normalized_viability < 0.7 ~ TRUE,
                                             TRUE ~ FALSE))) %>%
  mutate(mono_lymph_sum = mono_percent_fresh + lymph_percent_fresh)


is_date <- function(x, ...) !all(is.na(as.Date(as.character(x), ...)))

reference_file <- here("qmd/cd34_all_202209015.csv")

cd34_all <- read_csv(reference_file) %>%
  mutate(across(where(~ is_date(.x, format = "%m/%d/%y")), ~ as_date(.x, format = "%m/%d/%y"))) %>%
  mutate(poor_recovery = as_factor(case_when(normalized_viability < 0.7 ~ TRUE,
                                             TRUE ~ FALSE))) %>%
  mutate(across(where(is.character), as.factor)) %>% 
  mutate(mono_percent_fresh = as.numeric(mono_percent_fresh),
         across(c(patient_name, patient_mrn, crid_id, product_id), as.character),
         across(c(ethnicit, whyretx, mblzmore, agcsf, agmcsf, apgcsf, aplerixa, 
                  aotcxcr4, autochem, pbu, pbuunit, pbumeth, pcarb, pcarbuni,
                  pbcnu, pbcnuuni, pcisp, pcispuni, pcortico, pcy, pcyunit, parac,
                  paraunit, pvp16, pvp16uni, pfludara, pifos, pgleevec, pccnu,
                  plpam, plpamuni, pmito, pmab, pcampath, ptaxol, pvm26, pthio,
                  pthiouni, pothdrug, podrguni, pradnmab, rad, pkinetic, pclad,
                  psolumed, psolunit, pprednis, pdexmeth, pothcort, pitchemo, 
                  plpmeth, pnitro, potnitro, ptrosulf, ptyrkini), as.factor),
         patient_mrn = cgtr::fix_mrns(patient_mrn)) %>%
  mutate(across(where(is.factor), ~ fct_explicit_na(.x, "Unknown")))

Demographics Table (Table 2)

Create the gtsummary Demographics Table (Table 2) 
table_1 <- cd34_all %>% 
  transmute(
    sex, cytometer, poor_recovery
    , age = round(as.numeric(difftime(cryo_date, dob, units = "days"))/365, 1)
    , diagnosis = as.character(diagnosis)
    # Mobilization Vars
    # , agcsf, agmcsf, apgcsf, aplerixa, aotcxcr4, autochem
    , wash_recovery
    , prepbl_cd34, tnc_fresh, cd34_percent_fresh, hpc_cd45_brightness
    , pmn_percent_fresh, lymph_percent_fresh, mono_percent_fresh
    , mnc_percentage = lymph_percent_fresh + mono_percent_fresh
    , qc_date_diff, qc_tnc_fresh, viability_7aad_percent_thawed
    , cd34_percent_thawed
  ) %>% 
  # Optional for grouping rows in tbl_summary
  mutate(
    diagnosis = case_when(
      diagnosis %notin% c("NBL", "Medullo", "HD", "Wilms", "ATRT") ~ "Other",
      TRUE ~ diagnosis),
    diagnosis = factor(diagnosis, levels = c("NBL", "Medullo", "HD", "Wilms", "ATRT", "Other"))
  ) %>% 
  tbl_summary(
    by = poor_recovery,
    label = list(
      sex ~ "Sex"
      , age ~ "Age"
      , cytometer ~ "Cytometer"
      , diagnosis ~ "Diagnosis"
      , prepbl_cd34 ~ "Prepbl CD34"
      , tnc_fresh ~ "TNC Fresh"
      , wash_recovery ~ "Wash Recovery"
      , cd34_percent_fresh ~ "CD34 % Fresh"
      , hpc_cd45_brightness ~ "HCP CD45 Brightness"
      , pmn_percent_fresh ~ "PMN % Fresh"
      , lymph_percent_fresh ~ "Lymph % Fresh"
      , mono_percent_fresh ~ "Mono % Fresh"
      , mnc_percentage ~ "MNC %"
      , qc_date_diff ~ "QC Date Diff"
      , qc_tnc_fresh ~ "QC TNC Fresh"
      , viability_7aad_percent_thawed ~ "Viability 7 AAD Thawed"
      , cd34_percent_thawed ~ "CD34 % Thawed"
    )
  ) %>%
  add_p() %>% 
  modify_spanning_header(all_stat_cols() ~ "**Poor Recovery**") %>% 
  as_flex_table()

table_1

Poor Recovery

Characteristic

FALSE, N = 3071

TRUE, N = 631

p-value2

Sex

0.8

F

156 (51%)

31 (49%)

M

151 (49%)

32 (51%)

Cytometer

0.5

Calibur

133 (43%)

30 (48%)

MQuant

174 (57%)

33 (52%)

Age

6.1 (3.4, 11.9)

6.0 (2.1, 15.3)

0.9

Diagnosis

0.12

NBL

147 (48%)

20 (32%)

Medullo

52 (17%)

14 (22%)

HD

33 (11%)

13 (21%)

Wilms

13 (4.2%)

3 (4.8%)

ATRT

12 (3.9%)

3 (4.8%)

Other

50 (16%)

10 (16%)

Wash Recovery

94.0 (91.3, 98.0)

95.0 (91.0, 98.0)

>0.9

Prepbl CD34

0.17 (0.08, 0.35)

0.18 (0.09, 0.71)

0.2

Unknown

2

0

TNC Fresh

195 (144, 226)

213 (174, 226)

0.10

CD34 % Fresh

0.73 (0.33, 1.51)

0.92 (0.38, 2.42)

0.055

HCP CD45 Brightness

0.18 (0.15, 0.21)

0.15 (0.12, 0.17)

<0.001

PMN % Fresh

16 (5, 36)

16 (5, 38)

>0.9

Unknown

4

1

Lymph % Fresh

29 (19, 44)

24 (14, 41)

0.2

Unknown

4

1

Mono % Fresh

41 (25, 58)

42 (29, 55)

0.8

Unknown

4

1

MNC %

79 (60, 91)

79 (59, 91)

0.5

Unknown

4

1

QC Date Diff

6 (2, 14)

7 (2, 14)

0.3

Unknown

7

0

QC TNC Fresh

190 (141, 226)

209 (174, 222)

0.11

Viability 7 AAD Thawed

90 (83, 94)

89 (78, 94)

0.5

CD34 % Thawed

0.77 (0.35, 1.54)

0.46 (0.22, 1.21)

0.017

1n (%); Median (IQR)

2Pearson's Chi-squared test; Wilcoxon rank sum test; Fisher's exact test

Distribution of vCD34% post-thaw compared to fresh product histogram (Figure 2)

Create histogram of vCD34% comparison for visualization. Actual Figure 2 re-binned and plotted in Excel. 
cd34 %>% 
  mutate(
    Viability = case_when(
      normalized_viability < 0.5 ~ "< 50%",
      normalized_viability >= 0.5 & normalized_viability < 0.70 ~ "50 - < 70%",
      normalized_viability >= 0.70 & normalized_viability < 1 ~ "70 - < 100%",
      normalized_viability >= 1 ~ ">= 100%",
      TRUE ~ NA_character_
    ),
    normalized_viability = normalized_viability * 100,
    Viability = factor(Viability, levels = c("< 50%",
                                                           "50 - < 70%",
                                                           "70 - < 100%",
                                                           ">= 100%")
    )
  ) %>% 
  ggplot(aes(x = normalized_viability, fill = Viability)) +
  geom_histogram(bins = 65) +
  scale_fill_manual(values = c("#FE0000", "#FFFF01", "#2F5596", "#00AF50")) +
  ylim(c(0, 60)) +
  theme_minimal() +
  xlab("Percentage of vCD34 % Remaining after Freeze-Thaw") + ylab("Number of Products")

Rescale flow data

Rescale flow data of the legacy cytometer to the current cytometer 
calibur_mean <- mean(cd34$hpc_cd45_brightness[cd34$cytometer == "Calibur"])
calibur_sd <- sd(cd34$hpc_cd45_brightness[cd34$cytometer == "Calibur"])
mquant_sd <- sd(cd34$hpc_cd45_brightness[cd34$cytometer == "MQuant"])
mquant_mean <- mean(cd34$hpc_cd45_brightness[cd34$cytometer == "MQuant"])

cd34 <- cd34 %>% 
  mutate(
    hpc_cd45_brightness_normalized = case_when(
      cytometer == "Calibur" ~ (((hpc_cd45_brightness-calibur_mean)/calibur_sd)*mquant_sd)+mquant_mean,
      TRUE ~ hpc_cd45_brightness
    )
  )

Modeling

Set up data split and cross fold validation 
set.seed(100)
cd34_split <- initial_split(cd34, prop = .85, strata = poor_recovery)
cd34_train <- training(cd34_split)
cd34_test  <- testing(cd34_split)

cd34_folds <- vfold_cv(cd34_train, v = 10)
Create tidymodels recipe with feature engineering 
ignore_columns <- c(
  "normalized_viability", # continuous outcome variable
  "hpc_cd45_brightness" # we will use the rescaled value
)

cd34_recipe <- 
  recipe(poor_recovery ~ ., data = cd34_train) %>%
  update_role(all_of(ignore_columns), new_role = "ignore") %>% 
  step_log(hpc_cd45_brightness_normalized) %>% 
  step_impute_knn(all_numeric_predictors(), all_nominal_predictors()) %>%
  step_dummy(all_nominal_predictors(), keep_original_cols = FALSE) %>% 
  step_select(
    poor_recovery,
    hpc_cd45_brightness_normalized, 
    diagnosis_HD, # Most predictive signal in this diagnosis
    mono_lymph_sum
  ) %>% 
  step_interact(terms = ~ (all_predictors())^2)
Create models and tidymodels workflows 
rf_model <- 
  rand_forest() %>%
  set_engine("ranger") %>%
  set_mode("classification")

knn_model <-
  nearest_neighbor() %>%
  set_engine("kknn") %>%
  set_mode("classification")

xgb_model <-
  boost_tree() %>%
  set_engine("xgboost") %>%
  set_mode("classification")

rf_workflow <- 
  workflow() %>%
  add_recipe(cd34_recipe) %>%
  add_model(rf_model)
set.seed(100)
rf_workflow %>%
  fit_resamples(cd34_folds) %>%
  collect_metrics()
## # A tibble: 2 × 6
##   .metric  .estimator  mean     n std_err .config             
##   <chr>    <chr>      <dbl> <int>   <dbl> <chr>               
## 1 accuracy binary     0.831    10  0.0260 Preprocessor1_Model1
## 2 roc_auc  binary     0.729    10  0.0392 Preprocessor1_Model1

knn_workflow <- 
  rf_workflow %>%
  update_model(knn_model)
set.seed(100)
knn_workflow %>%
  fit_resamples(cd34_folds) %>%
  collect_metrics()
## # A tibble: 2 × 6
##   .metric  .estimator  mean     n std_err .config             
##   <chr>    <chr>      <dbl> <int>   <dbl> <chr>               
## 1 accuracy binary     0.818    10  0.0259 Preprocessor1_Model1
## 2 roc_auc  binary     0.741    10  0.0287 Preprocessor1_Model1

xgb_workflow <-
  rf_workflow %>%
  update_model(xgb_model)
set.seed(100)
xgb_workflow %>%
  fit_resamples(cd34_folds) %>%
  collect_metrics()
## # A tibble: 2 × 6
##   .metric  .estimator  mean     n std_err .config             
##   <chr>    <chr>      <dbl> <int>   <dbl> <chr>               
## 1 accuracy binary     0.818    10  0.0262 Preprocessor1_Model1
## 2 roc_auc  binary     0.735    10  0.0408 Preprocessor1_Model1
Tune the model 
xgb_tune_model <- list()
xgb_tune_results <- list()

xgb_tune_model[[1]] <- 
  boost_tree(learn_rate = 0.1,
             trees = tune()) %>%
  set_engine("xgboost") %>%
  set_mode("classification")

xgb_tune_workflow <-
  xgb_workflow %>%
  update_model(xgb_tune_model[[1]])  

set.seed(100)
xgb_tune_results[[1]] <- 
  xgb_tune_workflow %>%
  tune_grid(cd34_folds)

best_trees <- select_best(xgb_tune_results[[1]])$trees

xgb_tune_results[[1]] %>%
  autoplot()

Tune the model 

xgb_tune_model[[2]] <- 
  boost_tree(learn_rate = 0.1,
             trees = best_trees,
             tree_depth = tune()) %>%  
  set_engine("xgboost") %>%
  set_mode("classification")

xgb_tune_workflow <-
  xgb_workflow %>%
  update_model(xgb_tune_model[[2]]) 

set.seed(100)
xgb_tune_results[[2]] <- 
  xgb_tune_workflow %>%
  tune_grid(cd34_folds)

best_tree_depth <- select_best(xgb_tune_results[[2]])$tree_depth

xgb_tune_results[[2]] %>%
  autoplot()

Tune the model 

xgb_tune_model[[3]] <- 
  boost_tree(learn_rate = 0.1,
             trees = best_trees,
             tree_depth = best_tree_depth,
             mtry = tune()) %>% 
  set_engine("xgboost") %>%
  set_mode("classification")

xgb_tune_workflow <-
  xgb_workflow %>%
  update_model(xgb_tune_model[[3]]) 

set.seed(100)
xgb_tune_results[[3]] <- 
  xgb_tune_workflow %>%
  tune_grid(cd34_folds)

best_mtry <- select_best(xgb_tune_results[[3]])$mtry

xgb_tune_results[[3]] %>%
  autoplot()

Estimate a conservative classification threshold 
set.seed(100)
best_model <-
  xgb_tune_results[[3]] %>% 
  select_best(metric = "roc_auc")

final_workflow <- 
  xgb_tune_workflow %>%
  finalize_workflow(best_model)

assessment_set_predictions <- final_workflow %>%
  fit_resamples(cd34_folds, control = control_resamples(save_pred = TRUE)) %>%
  collect_predictions()
Evaluate the ROC curve (Supplemental Figure 3A) 
roc_values <-
  assessment_set_predictions %>%
  roc_curve(truth = poor_recovery, estimate = .pred_FALSE)
autoplot(roc_values) + 
  xlab("True Positive Rate") + 
  ylab("False Positive Rate")

Create confusion matrix with default probability threshold (Supplemental Figure 3B) 
truth_table <- function(data, truth, prediction) {
  data %>%
    conf_mat(truth = {{ truth }}, estimate = {{ prediction }}) %>%
    autoplot(type = "heatmap") +
    scale_x_discrete(limits = c(TRUE, FALSE), position = "top") + 
    scale_y_discrete(limits = c(FALSE, TRUE))
}

assessment_set_predictions %>%
  truth_table(truth = poor_recovery, prediction = .pred_class) + 
  xlab("Actual Poor vCD34 Recovery") +
  ylab("Predicted Poor vCD34 Recovery")

Create confusion matrix with sensitive probability threshold (Supplemental Figure 3C) 
threshold <- 0.003
assessment_set_predictions <- assessment_set_predictions %>%
  mutate(.pred_class = (.pred_TRUE > threshold) %>% as.factor)

assessment_set_predictions %>%
  truth_table(truth = poor_recovery, prediction = .pred_class) + 
  xlab("Actual Poor vCD34 Recovery") +
  ylab("Predicted Poor vCD34 Recovery")

Finalize the Model

Finalize model using XGBoost 
test_results <-
  final_workflow %>% 
  last_fit(split = cd34_split)

test_results %>% 
  collect_metrics()
## # A tibble: 2 × 4
##   .metric  .estimator .estimate .config             
##   <chr>    <chr>          <dbl> <chr>               
## 1 accuracy binary         0.877 Preprocessor1_Model1
## 2 roc_auc  binary         0.834 Preprocessor1_Model1
Collect predicitions 
cd34_predictions <- test_results %>%
  collect_predictions()
cd34_predictions
## # A tibble: 57 × 7
##    id               .pred_FALSE .pred_TRUE  .row .pred_class poor_reco…¹ .config
##    <chr>                  <dbl>      <dbl> <int> <fct>       <fct>       <chr>  
##  1 train/test split       0.851   0.149        5 FALSE       TRUE        Prepro…
##  2 train/test split       0.999   0.000867    12 FALSE       FALSE       Prepro…
##  3 train/test split       0.648   0.352       17 FALSE       FALSE       Prepro…
##  4 train/test split       0.999   0.00144     31 FALSE       FALSE       Prepro…
##  5 train/test split       0.999   0.000533    34 FALSE       FALSE       Prepro…
##  6 train/test split       0.961   0.0387      42 FALSE       FALSE       Prepro…
##  7 train/test split       0.927   0.0731      43 FALSE       FALSE       Prepro…
##  8 train/test split       0.998   0.00222     47 FALSE       FALSE       Prepro…
##  9 train/test split       0.359   0.641       73 TRUE        FALSE       Prepro…
## 10 train/test split       0.997   0.00309     77 FALSE       FALSE       Prepro…
## # … with 47 more rows, and abbreviated variable name ¹​poor_recovery
Evaluate final ROC curve (Figure 4A) 
roc_values <-
  cd34_predictions %>%
  roc_curve(truth = poor_recovery, estimate = .pred_FALSE)
autoplot(roc_values) + 
  xlab("True Positive Rate") + 
  ylab("False Positive Rate")

Create final confusion matrix (Figure 4B) 
threshold <- 0.003
cd34_predictions %>%
  mutate(.pred_class = (.pred_TRUE > threshold) %>% as.factor) %>%
  truth_table(truth = poor_recovery, prediction = .pred_class) + 
  xlab("Actual Poor vCD34 Recovery") +
  ylab("Predicted Poor vCD34 Recovery")

Check variable importance (Figure 4C) 
library(vip)
test_fit <- final_workflow %>% 
  fit(cd34_test) %>%
  pluck("fit", "fit")
vip(test_fit)

Computational Environment

Computational Environment 
devtools::session_info()
## ─ Session info ───────────────────────────────────────────────────────────────
##  setting  value
##  version  R version 4.2.0 (2022-04-22)
##  os       macOS Catalina 10.15.7
##  system   x86_64, darwin17.0
##  ui       X11
##  language (EN)
##  collate  en_US.UTF-8
##  ctype    en_US.UTF-8
##  tz       America/New_York
##  date     2022-09-15
##  pandoc   2.18 @ /Applications/RStudio.app/Contents/MacOS/quarto/bin/tools/ (via rmarkdown)
## 
## ─ Packages ───────────────────────────────────────────────────────────────────
##  package       * version    date (UTC) lib source
##  assertthat      0.2.1      2019-03-21 [1] CRAN (R 4.2.0)
##  backports       1.4.1      2021-12-13 [1] CRAN (R 4.2.0)
##  base64enc       0.1-3      2015-07-28 [1] CRAN (R 4.2.0)
##  bit             4.0.4      2020-08-04 [1] CRAN (R 4.2.0)
##  bit64           4.0.5      2020-08-30 [1] CRAN (R 4.2.0)
##  broom         * 1.0.0      2022-07-01 [1] CRAN (R 4.2.0)
##  broom.helpers   1.8.0      2022-07-05 [1] CRAN (R 4.2.0)
##  cachem          1.0.6      2021-08-19 [1] CRAN (R 4.2.0)
##  callr           3.7.1      2022-07-13 [1] CRAN (R 4.2.0)
##  cellranger      1.1.0      2016-07-27 [1] CRAN (R 4.2.0)
##  cgtr          * 0.1.7      2022-06-06 [1] CHOPRAN (R 4.2.0)
##  class           7.3-20     2022-01-16 [1] CRAN (R 4.2.0)
##  cli             3.4.0      2022-09-08 [1] CRAN (R 4.2.0)
##  codetools       0.2-18     2020-11-04 [1] CRAN (R 4.2.0)
##  colorspace      2.0-3      2022-02-21 [1] CRAN (R 4.2.0)
##  crayon          1.5.1      2022-03-26 [1] CRAN (R 4.2.0)
##  data.table      1.14.2     2021-09-27 [1] CRAN (R 4.2.0)
##  DBI             1.1.3      2022-06-18 [1] CRAN (R 4.2.0)
##  dbplyr          2.2.1      2022-06-27 [1] CRAN (R 4.2.0)
##  devtools        2.4.4      2022-07-20 [1] CRAN (R 4.2.0)
##  dials         * 1.0.0      2022-06-14 [1] CRAN (R 4.2.0)
##  DiceDesign      1.9        2021-02-13 [1] CRAN (R 4.2.0)
##  digest          0.6.29     2021-12-01 [1] CRAN (R 4.2.0)
##  dplyr         * 1.0.10     2022-09-01 [1] CRAN (R 4.2.0)
##  DT            * 0.24       2022-08-09 [1] CRAN (R 4.2.0)
##  ellipsis        0.3.2      2021-04-29 [1] CRAN (R 4.2.0)
##  evaluate        0.16       2022-08-09 [1] CRAN (R 4.2.0)
##  fansi           1.0.3      2022-03-24 [1] CRAN (R 4.2.0)
##  farver          2.1.1      2022-07-06 [1] CRAN (R 4.2.0)
##  fastmap         1.1.0      2021-01-25 [1] CRAN (R 4.2.0)
##  flextable     * 0.7.3      2022-08-09 [1] CRAN (R 4.2.0)
##  forcats       * 0.5.1      2021-01-27 [1] CRAN (R 4.2.0)
##  foreach         1.5.2      2022-02-02 [1] CRAN (R 4.2.0)
##  fs              1.5.2      2021-12-08 [1] CRAN (R 4.2.0)
##  furrr           0.3.0      2022-05-04 [1] CRAN (R 4.2.0)
##  future          1.27.0     2022-07-22 [1] CRAN (R 4.2.0)
##  future.apply    1.9.0      2022-04-25 [1] CRAN (R 4.2.0)
##  gargle          1.2.0      2021-07-02 [1] CRAN (R 4.2.0)
##  gdtools         0.2.4      2022-02-14 [1] CRAN (R 4.2.0)
##  generics        0.1.3      2022-07-05 [1] CRAN (R 4.2.0)
##  ggplot2       * 3.3.6      2022-05-03 [1] CRAN (R 4.2.0)
##  globals         0.16.0     2022-08-05 [1] CRAN (R 4.2.0)
##  glue            1.6.2      2022-02-24 [1] CRAN (R 4.2.0)
##  googledrive     2.0.0      2021-07-08 [1] CRAN (R 4.2.0)
##  googlesheets4   1.0.0      2021-07-21 [1] CRAN (R 4.2.0)
##  gower           1.0.0      2022-02-03 [1] CRAN (R 4.2.0)
##  GPfit           1.0-8      2019-02-08 [1] CRAN (R 4.2.0)
##  gridExtra       2.3        2017-09-09 [1] CRAN (R 4.2.0)
##  gt              0.6.0      2022-05-24 [1] CRAN (R 4.2.0)
##  gtable          0.3.0      2019-03-25 [1] CRAN (R 4.2.0)
##  gtsummary     * 1.6.1      2022-06-22 [1] CRAN (R 4.2.0)
##  hardhat         1.2.0      2022-06-30 [1] CRAN (R 4.2.0)
##  haven           2.5.0      2022-04-15 [1] CRAN (R 4.2.0)
##  here          * 1.0.1      2020-12-13 [1] CRAN (R 4.2.0)
##  hms             1.1.2      2022-08-19 [1] CRAN (R 4.2.0)
##  htmltools       0.5.3      2022-07-18 [1] CRAN (R 4.2.0)
##  htmlwidgets     1.5.4      2021-09-08 [1] CRAN (R 4.2.0)
##  httpuv          1.6.5      2022-01-05 [1] CRAN (R 4.2.0)
##  httr            1.4.4      2022-08-17 [1] CRAN (R 4.2.0)
##  igraph          1.3.4      2022-07-19 [1] CRAN (R 4.2.0)
##  infer         * 1.0.2      2022-06-10 [1] CRAN (R 4.2.0)
##  ipred           0.9-13     2022-06-02 [1] CRAN (R 4.2.0)
##  iterators       1.0.14     2022-02-05 [1] CRAN (R 4.2.0)
##  jsonlite        1.8.0      2022-02-22 [1] CRAN (R 4.2.0)
##  kknn          * 1.3.1      2016-03-26 [1] CRAN (R 4.2.0)
##  knitr           1.39       2022-04-26 [1] CRAN (R 4.2.0)
##  labeling        0.4.2      2020-10-20 [1] CRAN (R 4.2.0)
##  later           1.3.0      2021-08-18 [1] CRAN (R 4.2.0)
##  lattice         0.20-45    2021-09-22 [1] CRAN (R 4.2.0)
##  lava            1.6.10     2021-09-02 [1] CRAN (R 4.2.0)
##  lhs             1.1.5      2022-03-22 [1] CRAN (R 4.2.0)
##  lifecycle       1.0.2      2022-09-09 [1] CRAN (R 4.2.0)
##  listenv         0.8.0      2019-12-05 [1] CRAN (R 4.2.0)
##  lubridate     * 1.8.0      2021-10-07 [1] CRAN (R 4.2.0)
##  magrittr        2.0.3      2022-03-30 [1] CRAN (R 4.2.0)
##  MASS            7.3-58.1   2022-08-03 [1] CRAN (R 4.2.0)
##  Matrix          1.4-1      2022-03-23 [1] CRAN (R 4.2.0)
##  memoise         2.0.1      2021-11-26 [1] CRAN (R 4.2.0)
##  mime            0.12       2021-09-28 [1] CRAN (R 4.2.0)
##  miniUI          0.1.1.1    2018-05-18 [1] CRAN (R 4.2.0)
##  modeldata     * 1.0.0      2022-07-01 [1] CRAN (R 4.2.0)
##  modelr          0.1.8      2020-05-19 [1] CRAN (R 4.2.0)
##  munsell         0.5.0      2018-06-12 [1] CRAN (R 4.2.0)
##  nnet            7.3-17     2022-01-16 [1] CRAN (R 4.2.0)
##  officer         0.4.3      2022-06-12 [1] CRAN (R 4.2.0)
##  parallelly      1.32.1     2022-07-21 [1] CRAN (R 4.2.0)
##  parsnip       * 1.0.0      2022-06-16 [1] CRAN (R 4.2.0)
##  pillar          1.8.1      2022-08-19 [1] CRAN (R 4.2.0)
##  pkgbuild        1.3.1      2021-12-20 [1] CRAN (R 4.2.0)
##  pkgconfig       2.0.3      2019-09-22 [1] CRAN (R 4.2.0)
##  pkgload         1.3.0      2022-06-27 [1] CRAN (R 4.2.0)
##  prettyunits     1.1.1      2020-01-24 [1] CRAN (R 4.2.0)
##  processx        3.7.0      2022-07-07 [1] CRAN (R 4.2.0)
##  prodlim         2019.11.13 2019-11-17 [1] CRAN (R 4.2.0)
##  profvis         0.3.7      2020-11-02 [1] CRAN (R 4.2.0)
##  promises        1.2.0.1    2021-02-11 [1] CRAN (R 4.2.0)
##  ps              1.7.1      2022-06-18 [1] CRAN (R 4.2.0)
##  purrr         * 0.3.4      2020-04-17 [1] CRAN (R 4.2.0)
##  R6              2.5.1      2021-08-19 [1] CRAN (R 4.2.0)
##  ranger        * 0.14.1     2022-06-18 [1] CRAN (R 4.2.0)
##  Rcpp            1.0.9      2022-07-08 [1] CRAN (R 4.2.0)
##  readr         * 2.1.2      2022-01-30 [1] CRAN (R 4.2.0)
##  readxl          1.4.0      2022-03-28 [1] CRAN (R 4.2.0)
##  recipes       * 1.0.1      2022-07-07 [1] CRAN (R 4.2.0)
##  REDCapR         1.1.0      2022-08-10 [1] CRAN (R 4.2.0)
##  remotes         2.4.2      2021-11-30 [1] CRAN (R 4.2.0)
##  reprex          2.0.1      2021-08-05 [1] CRAN (R 4.2.0)
##  rJava         * 1.0-6      2021-12-10 [1] CRAN (R 4.2.0)
##  RJDBC           0.2-10     2022-03-24 [1] CRAN (R 4.2.0)
##  rlang           1.0.5      2022-08-31 [1] CRAN (R 4.2.0)
##  rmarkdown       2.14       2022-04-25 [1] CRAN (R 4.2.0)
##  ROSE            0.0-4      2021-06-14 [1] CRAN (R 4.2.0)
##  rpart           4.1.16     2022-01-24 [1] CRAN (R 4.2.0)
##  rprojroot       2.0.3      2022-04-02 [1] CRAN (R 4.2.0)
##  rsample       * 1.1.0      2022-08-08 [1] CRAN (R 4.2.0)
##  rstudioapi      0.13       2020-11-12 [1] CRAN (R 4.2.0)
##  rvest           1.0.2      2021-10-16 [1] CRAN (R 4.2.0)
##  scales        * 1.2.0      2022-04-13 [1] CRAN (R 4.2.0)
##  sessioninfo     1.2.2      2021-12-06 [1] CRAN (R 4.2.0)
##  shiny           1.7.2      2022-07-19 [1] CRAN (R 4.2.0)
##  stringi         1.7.8      2022-07-11 [1] CRAN (R 4.2.0)
##  stringr       * 1.4.1      2022-08-20 [1] CRAN (R 4.2.0)
##  survival        3.4-0      2022-08-09 [1] CRAN (R 4.2.0)
##  systemfonts     1.0.4      2022-02-11 [1] CRAN (R 4.2.0)
##  themis        * 1.0.0      2022-07-02 [1] CRAN (R 4.2.0)
##  tibble        * 3.1.8      2022-07-22 [1] CRAN (R 4.2.0)
##  tidymodels    * 1.0.0      2022-07-13 [1] CRAN (R 4.2.0)
##  tidyr         * 1.2.1      2022-09-08 [1] CRAN (R 4.2.0)
##  tidyselect      1.1.2      2022-02-21 [1] CRAN (R 4.2.0)
##  tidyverse     * 1.3.2      2022-07-18 [1] CRAN (R 4.2.0)
##  timeDate        4021.104   2022-07-19 [1] CRAN (R 4.2.0)
##  tune          * 1.0.0      2022-07-07 [1] CRAN (R 4.2.0)
##  tzdb            0.3.0      2022-03-28 [1] CRAN (R 4.2.0)
##  urlchecker      1.0.1      2021-11-30 [1] CRAN (R 4.2.0)
##  usethis         2.1.6      2022-05-25 [1] CRAN (R 4.2.0)
##  utf8            1.2.2      2021-07-24 [1] CRAN (R 4.2.0)
##  uuid            1.1-0      2022-04-19 [1] CRAN (R 4.2.0)
##  vctrs           0.4.1      2022-04-13 [1] CRAN (R 4.2.0)
##  vip           * 0.3.2      2020-12-17 [1] CRAN (R 4.2.0)
##  visdat        * 0.5.3      2019-02-15 [1] CRAN (R 4.2.0)
##  vroom           1.5.7      2021-11-30 [1] CRAN (R 4.2.0)
##  withr           2.5.0      2022-03-03 [1] CRAN (R 4.2.0)
##  workflows     * 1.0.0      2022-07-05 [1] CRAN (R 4.2.0)
##  workflowsets  * 1.0.0      2022-07-12 [1] CRAN (R 4.2.0)
##  xfun            0.32       2022-08-10 [1] CRAN (R 4.2.0)
##  xgboost       * 1.6.0.1    2022-04-16 [1] CRAN (R 4.2.0)
##  xml2            1.3.3      2021-11-30 [1] CRAN (R 4.2.0)
##  xtable          1.8-4      2019-04-21 [1] CRAN (R 4.2.0)
##  yaml            2.3.5      2022-02-21 [1] CRAN (R 4.2.0)
##  yardstick     * 1.0.0      2022-06-06 [1] CRAN (R 4.2.0)
##  zip             2.2.0      2021-05-31 [1] CRAN (R 4.2.0)
## 
##  [1] /Library/Frameworks/R.framework/Versions/4.2/Resources/library
## 
## ──────────────────────────────────────────────────────────────────────────────