CMV viral load kinetics as surrogate endpoints after allogeneic transplantation

Despite advances in the treatment and prevention of CMV complications after hematopoietic cell transplantation (HCT), CMV remains an important cause of morbidity and mortality. CMV viremia is associated with increased nonrelapse mortality (1, 2), acute graft versus host disease (aGVHD) (3), and secondary bacterial and fungal infections (4, 5). Since the 1990s, HCT recipients have been treated preemptively with antiviral drugs for the prevention of CMV disease. However, all antivirals approved for the treatment of CMV disease and for preemptive therapy (ganciclovir, valganciclovir, and foscarnet) cause significant toxicities, including neutropenia, renal failure, and genital ulcers (6, 7). Additional antiviral therapies are needed to reduce CMV-related complications after HCT (6). Establishing CMV viral load–based (VL-based) surrogate endpoints for use in clinical trials would facilitate development of new antiviral therapeutics (7, 8). Indeed, the well-tolerated, antiviral drug letermovir was recently approved for preventing CMV reactivation as prophylactic therapy using a combined endpoint that included clinically significant infection (CMV VL at a level high enough to warrant preemptive therapy) (9).

Surrogate endpoints are biomarkers that predict clinical outcomes accurately enough to replace those outcomes in clinical trials. Using surrogate endpoints in clinical trials can reduce follow-up time and the number of patients required to demonstrate an effect, reducing research costs and burden to trial participants and facilitating delivery of new therapies to bedside (10, 11). FDA approval of VL surrogate endpoints revolutionized antiviral drug development for HIV and hepatitis C (8, 10). Clinical trials for HIV and hepatitis C now use VL-based endpoints, a practice that has dramatically reduced times to licensure of new antivirals (12–14).

HIV and hepatitis C VL surrogates were validated via meta-analyses of large numbers of randomized controlled trials (RCTs) that were performed during the era of VL testing with PCR (15, 16). A recent meta-analysis in solid organ transplantation has provided evidence that VL may be a valid surrogate for CMV disease in the solid organ transplantation setting, but lacks placebo-controlled RCT data (8). In the HCT setting, prior to our study, no VL data from placebo-controlled, randomized treatment trials existed because these trials were conducted long before the availability of PCR testing (17–21). Validating VL-based surrogates for CMV is not possible in the modern clinical environment because of the absence of placebo-controlled antiviral trials (for equipoise, ganciclovir and foscarnet are used as active controls based on their proven association with clinical benefit) and the small numbers of clinical CMV disease cases. However, despite changes in HCT care, CMV viral reactivation continues to occur in the modern setting and likely remains the primary mechanism through which CMV disease occurs (1, 2). To address whether VL-based surrogates are valid surrogate endpoints, we performed VL testing of frozen samples obtained during a historic clinical trial — the first and only double-blind, placebo-controlled, randomized trial for the early treatment of CMV infection with ganciclovir after bone marrow transplantation (BMT) — and calculated CMV VL kinetics to assess their potential use as surrogate endpoints (17).

We employed traditional statistical methods and state-of-the-art machine learning techniques to validate VL as a surrogate endpoint. The Prentice definition (traditional methodology) is a rigorous statistical standard for evaluating whether an intermediate response endpoint is a valid surrogate endpoint (11, 22, 23). We applied the Prentice definition to our data to evaluate whether VL kinetics could serve as valid surrogate endpoints and quantified the degree to which they captured ganciclovir’s effect on clinical outcomes. In addition, we employed ensemble-based machine learning models (Super Learner, ref. 24) to determine the ability of VL kinetics to predict clinical outcomes. Finally, given that many centers now use prophylaxis as their primary CMV prevention strategy, we used these same techniques to validate our results in the prophylactic setting with patient samples from the first ganciclovir prophylaxis RCT (18).

Ganciclovir reduced CMV disease and mortality at least 20 years after the original RCT. In a single-center study performed at the Fred Hutchinson Cancer Research Center from 1989 to 1990 (17), 72 allogeneic HCT recipients who were either CMV seropositive or who had received marrow from CMV seropositive donors were screened weekly for CMV with viral cultures and were randomized to receive either ganciclovir or a placebo at the time of first positive culture. A description of the study design is provided in Figure 1, A and B, and baseline patient characteristics are shown in Supplemental Table 1; supplemental material available online with this article; https://doi.org/10.1172/JCI133960DS1 A schematic of the VL analysis is shown in Figure 1C.

Figure 1

CONSORT diagram and study designs for the early treatment trial. Study design for Goodrich et al. RCT (A and B) and for VL kinetic analysis (C). (A) The reconstructed CONSORT diagram for the original RCT. (B) The original study design with surveillance and screening beginning at HCT and randomization beginning at the time of first positive surveillance culture. (C) The VL kinetics study design with analysis beginning at randomization (receipt of study drug) and ending at day 100 after HCT or a study endpoint of CMV disease or death, whichever occurred first.

The original trial was designed to enroll 116 patients, but it was terminated early after the interim analysis showed a large reduction in tissue-invasive CMV disease by 100 days after HCT. Ganciclovir was found to have reduced significantly the cumulative incidence of CMV disease and overall mortality at 100 and 180 days after HCT (Figure 2A). Extending the follow-up of results observed in the original RCT through chart review, we found that the cumulative incidence of CMV disease and of the composite endpoint of CMV disease or death remained significantly lower in the ganciclovir group after 20 years (Figure 2B). Overall mortality was also lower in the ganciclovir group after 20 years (Figure 2B), although the trend in mortality was no longer statistically significant by 10 years. When outcomes were counted from randomization rather than transplantation, the results were similar (Supplemental Figure 1). Detailed methods and results from the original study and extended follow-up are included in the supplemental material. By providing evidence of a successful intervention in an RCT, these results demonstrate that the Prentice definition can be applied to our data.

Figure 2

CMV disease and death clinical outcomes in the early treatment trial. CMV disease (right-censored for death), overall mortality, and first event of CMV disease or mortality in the placebo and ganciclovir (GCV) groups at time points defined in the original study (A) and at extended follow-up times out to 20 years (B). In all plots, the ganciclovir group is shown in red; the placebo group is shown in blue. Numbers at risk are shown below their respective plots (PLAC indicates the placebo group. GCV indicates the treatment group). Survival and first event of CMV disease or death curves were estimated using Kaplan-Meier methods. The cumulative incidence of CMV disease with death as a competing risk was estimated using the Aalen-Johnson method. Survival distributions and times to the composite endpoint of CMV disease or death were compared using a log-rank test. Cumulative incidence distributions for CMV disease with death as a competing risk were compared using Gray’s test.

Ganciclovir lowered CMV VL kinetics in the first 5 weeks after randomization. Validation of surrogate endpoints requires the measurement of candidate biomarkers at intermediate time points after randomization. Frozen serum samples left over from clinical testing were stored prospectively in a biorepository at Fred Hutchinson Cancer Research Center for all study participants. CMV DNA PCR VLs were measured from available samples collected at approximately weekly intervals up to day 100 after HCT. VLs collected near the time of randomization until the first event of CMV disease were included in the surrogate analysis. All 72 patients had VL samples available near the time of randomization. Sixty-five patients had at least 1 VL measured in weeks 1 through 5, and there was a median of 4 measurements per patient in both treatment groups. Detailed sample availability information is provided in Supplemental Table 2.

CMV VL kinetics, including mean VL (log₁₀ IU/mL), maximum change in VL from randomization (log₁₀ IU/mL), peak VL (log₁₀ IU/mL), and percentage of positive VLs (viral shedding rate) were calculated from VLs measured in the first 5 weeks after randomization. Only early VLs (weeks 1–5) were included because surrogate endpoints are more useful when measured early after interventions and because many patients in the placebo group died or developed CMV disease soon after randomization. Weekly mean VLs, changes in VL, and all VL kinetics are shown in Figure 3.

Figure 3

Weekly CMV VL kinetics in the early treatment trial. CMV VL kinetics from time of randomization (Week 0). (A and B) VL data are shown for patients who had not reached an endpoint of CMV disease or death by that week. GCV indicates patients in the ganciclovir treatment group (shown in red). Placebo indicates patients in the placebo treatment group (shown in blue). Error bars indicate 95% CI. The dashed horizontal line represents the limit of detection (LOD) of the CMV VL assay. VL kinetics summary calculations (C) were performed with the data shown in A and B. Box-and-whisker plots show the middle 50% of VL kinetics in gray boxes with a horizontal black line at the median. Whiskers extend upward from the third quartile at the top of the box to 1.5 times the IQR (the distance between first and third quartiles) and downward from the first quartile at the bottom of the box to 1.5 times the IQR. P values were calculated from 2-tailed t tests comparing the means of the viral kinetics in GCV versus placebo groups.

CMV VL kinetics fulfill the Prentice definition for valid surrogate endpoints. We evaluated whether each of the 4 VL kinetics defined above (mean, maximum change, peak, and percentage positive) is a valid surrogate for the clinical endpoints of tissue-invasive CMV disease or the composite endpoint of CMV disease or death by 8, 24, and 48 weeks after randomization (Figure 1C). In the main text, we focus on results for week 48. Results for weeks 8 and 24 are provided in Supplemental Results. Because patients were randomized at varying times from HCT based on positive viral culture results, we chose weeks rather than days to describe outcomes in the surrogate analysis to help differentiate time from randomization rather than time from transplantation.

We validated each VL kinetic based on fulfillment of the Prentice definition for valid surrogate endpoints. The Prentice definition requires that a hypothesis test of the treatment effect (e.g., ganciclovir effect) on the surrogate endpoint (e.g., VL kinetic) is a valid hypothesis test of the treatment effect on the clinical endpoint (e.g., CMV disease). In other words, if a clinical trial assessing the effect of a treatment was performed with the primary outcome being an effect on the surrogate marker rather than the clinical outcome, the overall conclusion would be the same as a study performed using the clinical endpoint (22, 23).

Prentice criterion 1. To satisfy the Prentice definition, surrogates must fulfill 3 main criteria. The first criterion requires that treatment (ganciclovir) affects the candidate surrogate endpoint (e.g., peak VL). This criterion was met for all VL kinetics as reported above and in Figure 3C, in that mean, maximum change, peak, and percentage of positive VLs were significantly lower in the ganciclovir group.

Prentice criterion 2. The second Prentice criterion is met if there is an association between candidate surrogates (VL kinetics) and clinical outcomes (CMV disease or death). Logistic regression models adjusted for aGVHD, CMV donor serostatus, and VL at randomization, but not for treatment group assignment, demonstrated that all VL kinetics met this criterion for all clinical endpoints at weeks 8, 24, and 48 (Supplemental Table 3), i.e., higher values of the VL kinetics correlated with significantly higher odds of CMV disease or death.

Prentice criterion 3. The third Prentice criterion states that for a given value of the candidate surrogate (e.g., maximum change in VL), the probability of the clinical outcome (e.g., CMV disease) is the same in each treatment group (ganciclovir or placebo group). We tested for fulfillment of this criterion with logistic regression models adjusted for aGVHD, CMV donor serostatus, VL at randomization, and treatment group. Because we adjusted for treatment group assignment and VL kinetics in these models, we were able to determine whether the treatment group assignment correlated with outcomes after adjustment for the VL kinetic. Thus, to fulfill Prentice criterion 3, the OR for the VL kinetic should be significantly greater than 1 at the P = 0.05 level, and the OR for the treatment assignment should not differ significantly from 1 at the P = 0.2 level (P value threshold higher to demonstrate similarity in values rather than difference). Figure 4 illustrates with asterisks that mean VL, peak VL, and maximum change in VL met Prentice criterion 3 (P < 0.05 for VL association; P ≥ 0.20 for treatment group association) for CMV disease by week 48 with no evidence of a treatment by marker interaction (P ≥ 0.20). Percentage of positive VLs nearly satisfied Prentice criteria (P = 0.07 for VL association). Mean, peak, and percentage of positive VLs also satisfied Prentice criteria for the composite outcome by week 48. Maximum change in VL did not meet Prentice criterion 3 for the composite outcome (P = 0.14 for treatment group association). Results for clinical endpoints occurring by weeks 8 and 24 were similar and are shown in Supplemental Figure 2.

Figure 4

Prentice criteria evaluation using multivariate logistic regression and proportion of treatment effect captured in the early treatment trial. (A) Forest plots of the ORs for associations of VL kinetics with risk for CMV disease and CMV disease or death by week 48 after randomization were calculated from logistic regression models adjusted for baseline characteristics and treatment group. OR for VL kinetics are indicated by navy dots surrounded by 95% CI indicated with navy lines; OR with 95% CI for treatment group assignment are shown with light green dots and lines. Asterisks (*) indicate VL kinetics that met the Prentice criteria by multivariable logistic regression testing, i.e. the coefficient for VL kinetic was significantly different from 0 (P < 0.05), whereas the treatment group assignment coefficient was not significantly different from 0 (P ≥ 0.20). The treatment by marker interaction coefficient was not significantly different from 0 (P ≥ 0.20) for any kinetic. The percentage positive did not meet Prentice criteria for CMV disease with P = 0.07 for VL kinetic association. Max change did not meet Prentice criteria for CMV disease with P = 0.14 for GCV association. For mean, max change, and peak, ORs were calculated as the ratio of odds of the clinical outcome in groups differing by log₁₀ IU/mL. For percentage positive, the OR was calculated as the ratio of odds of the clinical outcome in groups differing by 25% in percentage of samples with detectable VL. Dashed vertical lines indicate OR = 1. (B) The percentages of ganciclovir’s effect on clinical outcomes captured by the candidate surrogate were calculated using Kobayashi and Kuroki’s measure (23) and are shown for each of the VL kinetics.

VL kinetics capture a large proportion of ganciclovir’s effect on clinical outcomes. We quantified how much of ganciclovir’s effect on clinical outcomes could be attributed to its effects on VL kinetics using the proportion of treatment effect captured by candidate surrogate endpoints (23). For the week 48 clinical outcome of CMV disease, several VL kinetics captured nearly all of the effect of ganciclovir: mean (99.9%), change (96.6%), peak (98.5%), and percentage positive (95.8%) (Figure 4B). Mean, maximum change, and percentage positive captured at least 93% of ganciclovir’s effect on the composite outcome of CMV disease or death at week 48, whereas peak captured 84.5% (Figure 4B). Almost all VL kinetics were considered “moderate” (> 63%) or “substantial” (> 85%) for composite outcomes by weeks 8 and 24. Maximum change captured 83.5% of ganciclovir’s effect on CMV disease by week 8, but other kinetics did not perform well for CMV disease by weeks 8 and 24 (Supplemental Figure 3B).

Super Learner predicts clinical outcomes with high accuracy. The Super Learner is a cross-validation–based ensemble machine learning method for estimating the optimal weighted average of the predictions from a library of algorithms. Each of these algorithms estimates the conditional probability of an event (e.g., CMV disease or no CMV disease) given a set of potential risk factors (e.g., VL kinetics or baseline risk factors) using cross-validation (24, 25). For surrogate validation, in addition to providing optimal prediction accuracy, Super Learner predictions have the advantage of evaluating the ability of surrogate endpoints to predict clinical outcomes for individuals, rather than describing mean behavior on the population level (26). We built Super Learner models using baseline covariates (aGVDH, CMV donor serostatus, and VL at randomization) and all VL kinetics (mean, maximum change, peak, percentage positive). As an exploratory analysis, we also fit Super Learner models using absolute lymphocyte kinetics.

We constructed receiver operating characteristic (ROC) curves to evaluate the sensitivity and specificity of Super Learner predictions for clinical outcomes and assessed their performance with leave-one-out cross-validated area under the ROC curves (cv-AUCs). cv-AUCs can be interpreted as the probability that a randomly selected patient experiencing a clinical outcome will have a higher predicted risk than a randomly selected patient not experiencing the outcome. Models that predict at the same level of accuracy as random chance have cv-AUC equal to 50%. Super Learner model predictions of both week 48 clinical outcomes yielded cv-AUCs greater than 90% (Figure 5, A–D). All models built on mean, maximum change, peak, and percentage positive VLs, whether fit separately on treatment groups or on the combined data set, predicted both clinical outcomes (CMV disease/CMV disease or death) at all time points (weeks 8, 24, and 48) with better than 85% cv-AUCs (Supplemental Figure 3A). Our results suggest that VL kinetics measured during the first 5 weeks of antiviral treatment combined with an ensemble machine learning algorithm allow for excellent clinical outcome prediction. In addition, models built on the placebo, ganciclovir, and combined groups performed similarly, consistent with the Prentice definition.

Figure 5

Prediction accuracy for clinical outcomes with Super Learner in the early treatment trial. (A and C) Receiver operating characteristic (ROC) curves are shown for Super Learner predictions for CMV disease and CMV disease or death by 48 weeks after randomization. The diagonal line drawn at y = x indicates the boundary above which ROC curves describe a prediction that is better than chance. (B and D) Forest plots show cross-validated area under the ROC curves (cv-AUC) of Super Learner predictions for CMV disease and CMV disease or death. For A–D, predictions made only on data from the placebo group are in blue, from the ganciclovir group (GCV) in red, and from both treatment groups (All) in purple. In B and D, the vertical line indicates cv-AUC = 50%, the area under the diagonal line in A and C.

To evaluate the contributions of VL kinetics to the accuracy of the Super Learner predictions, we fit Super Learner models using baseline characteristics only versus baseline characteristics plus all VL kinetics. We found that adding all VL kinetics to the baseline characteristics increased prediction accuracy greatly for all time points and both clinical outcomes (Supplemental Figure 4). For example, the model built on baseline characteristics alone had a cv-AUC of 75.5% (95% CI, 61%–90%) for CMV disease or death by week 8, but the cv-AUC increased to 96.8% (95% CI, 93%–100%) when VL kinetics were included.

Including absolute lymphocyte counts in Super Learner models improves prediction of some clinical outcomes. We calculated absolute lymphocyte count (ALC) kinetics, including ALC peak, ALC nadir, and mean ALC, during the 5-week period after randomization to explore whether adding longitudinal measures of immunity to the machine learning models might improve prediction accuracy for clinical outcomes (27). In addition, we added ALC at randomization to the baseline risk characteristics (donor CMV serostatus, aGVHD, and VL at randomization). We found that adding ALC kinetics did not change the prediction accuracy of CMV disease by earlier time points (weeks 8 and 24), but improved prediction of CMV disease by week 48. ALC also improved prediction of CMV disease or death at all time points (Supplemental Figure 5). However, importantly, absolute lymphocyte kinetics did not consistently increase or decrease with ganciclovir administration (Supplemental Figure 6), and thus cannot be assessed as surrogates for antiviral treatment. A surrogate of treatment effect must be affected in a consistent direction by the intervention.

Validation analysis performed from the ganciclovir prophylaxis RCT demonstrates VL kinetics are valid surrogates in the prophylaxis setting. As follow-up to the early treatment trial, ganciclovir was studied as a prophylactic agent in a placebo-controlled RCT at the Fred Hutchinson Cancer Research Center from 1990 to 1991. Sixty-four CMV-seropositive allogeneic HCT recipients were randomized to receive ganciclovir or a placebo at engraftment and were followed for development of CMV infection (by culture) and CMV disease (18). The CONSORT diagram and trial design schematic are shown in Figure 6, A and B. Baseline patient characteristics are shown in Supplemental Table 4. We analyzed clinical outcomes by weeks 14, 24, and 48. The cumulative incidence of CMV disease was significantly lower in the ganciclovir treatment group by weeks 14 and 24, but no difference in mortality was found at these or later time points (Figure 7). The same results are shown in Supplemental Figure 7 in days and years from transplant rather than randomization. The same VL kinetics, mean, peak, maximum change, and percentage of positive VLs (shedding rate), were calculated for the first 5 weeks after randomization. As in the early treatment RCT, all VL kinetics were significantly lower in the ganciclovir group, fulfilling Prentice criterion 1 (Figure 8).

Figure 6

CONSORT diagram and study design for the prophylaxis trial. Study design for Goodrich et al. AIM 1993 RCT. (A) The reconstructed CONSORT diagram for the original RCT. (B) The original study design with surveillance and screening beginning at HCT and randomization beginning at the time of engraftment.

Figure 7

CMV disease clinical outcomes in the prophylaxis trial. CMV disease (right-censored for death), overall mortality, and first event of CMV disease or mortality in the placebo and ganciclovir groups at 14, 24, and 48 weeks after randomization. The ganciclovir group is shown in red; the placebo group is shown in blue. Numbers at risk are shown below their respective plots (PLAC indicates the placebo group. GCV indicates the treatment group). Survival and first event of CMV disease or death curves were estimated using Kaplan-Meier methods. The cumulative incidence of CMV disease with death as a competing risk was estimated using the Aalen-Johnson method. Survival distributions and times to the composite endpoint of CMV disease or death were compared using a log-rank test. Cumulative incidence distributions for CMV disease with death as a competing risk were compared using Gray’s test.

Figure 8

Weekly CMV VL kinetics in the prophylaxis trial. CMV VL kinetics from time of randomization (week 0). (A and B) VL data are shown for patients who had not reached an endpoint of CMV disease or death by that week. GCV indicates patients in the ganciclovir treatment group (shown in red). Placebo indicates patients in the placebo treatment group (shown in blue). Error bars indicate 95% CI. The dashed horizontal line represents the limit of detection (LOD) of the CMV VL assay. (C) VL kinetics summary calculations were performed with the data shown in A and B. Box-and-whisker plots show the middle 50% of VL kinetics in gray boxes with a horizontal black line at the median. Whiskers indicate 1.5 times the IQR of the VL kinetics. P values were calculated from 2-tailed t tests comparing the means of the viral kinetics in ganciclovir (GCV) versus placebo groups.

Because no CMV disease events occurred in the treatment group during the first 14 weeks of the study, we were unable to perform the analyses at this time point. Thus, CMV disease by week 24 served as our primary clinical outcome. Prentice criterion 2 was met for all VL kinetics by week 24 (Supplemental Table 5). Only the percentage of positive VLs (shedding rate) met Prentice criterion 3, demonstrating a significant association between VL after adjustment for treatment group (Figure 9A). However, the remaining VL kinetics, mean, peak, and maximum change, nearly fulfilled this criterion with OR 2.4, 95% CI (1.0–6.7), P = 0.07 for mean; OR 1.7, 95% CI (1.0–3.2), P = 0.06 for peak; and OR 1.7, 95% CI (1.0–3.2), P = 0.06 for maximum change in VL. Also, CMV VL kinetics captured a large percentage of ganciclovir’s effect by week 24 — mean captured 86.3%, peak 82.7%, maximum change 94.5%, and shedding rate 93.8% (Figure 9B). Super Learner models built using baseline characteristics of aGVHD, donor CMV serostatus, and baseline VL plus all VL kinetics as in the main analysis were able to predict CMV disease by week 24 with cv-AUCs greater than 75% (Figure 9, C and D). The results of this validation procedure support not only the robustness of our findings that VL kinetics can serve as surrogate endpoints for clinical outcomes under different treatment settings, but also the applicability to the modern antiviral prophylaxis setting.

Figure 9

Prentice criteria evaluation using multivariate logistic regression, proportion of treatment effect captured, and prediction accuracy for clinical outcomes with Super Learner in the prophylaxis trial. (A) Forest plots of the OR for associations of VL kinetics with risk for CMV disease by week 24 after randomization were calculated from logistic regression models adjusted for baseline characteristics and treatment group. ORs for VL kinetics are indicated by navy dots surrounded by 95% CI indicated with navy lines; OR with 95% CI for treatment group assignment shown with light green dots and lines. Asterisks (*) indicate VL kinetics that met the Prentice criteria by multivariable logistic regression testing. The dashed vertical line indicates OR = 1. (B) The percentages of ganciclovir’s effect on clinical outcomes captured by the candidate surrogate were calculated using Kobayashi and Kuroki’s measure (23) and are shown for each of the VL kinetics indicated. (C) Receiver operating characteristic (ROC) curves are shown for Super Learner predictions for CMV disease by week 24 after randomization. The diagonal line drawn at y = x indicates the boundary above which ROC curves describe a prediction that is better than chance. (D) The Forest plot shows cross-validated area under the ROC curves (cv-AUC) of Super Learner predictions for CMV disease. The vertical line indicates cv-AUC = 50%, the area under the diagonal line in C. For C and D, predictions made only on data from the placebo group are in blue, from the ganciclovir group (GCV) in red, and from both treatment groups (All) in purple.

Our study analyzes VL kinetics as surrogates for CMV clinical outcomes after HCT based on data from 2 randomized, placebo-controlled trials with highly successful interventions (early treatment and prophylaxis with ganciclovir). Because the data were obtained from placebo-controlled RCTs, we were able to apply the Prentice definition and consequently demonstrated that CMV VL kinetics may be valid surrogates for CMV disease or death after HCT. Several VL kinetics captured greater than 90% of ganciclovir’s clinical effects in both treatment and prophylaxis trials. Our analysis with Super Learner showed that VL kinetics could be used to make highly sensitive and specific predictions of clinical outcomes. In addition, in both trials, Super Learner predictions had similar accuracy when built on placebo or ganciclovir group VL kinetics, providing additional support for the Prentice analysis. To our knowledge, our study is the first to harness the power of machine learning to evaluate virological outcomes as surrogate endpoints. Likewise, the percentage of antiviral treatment effect captured by CMV VL has not been estimated previously.

In this study, we used modern laboratory testing and statistical techniques to analyze frozen samples from CMV treatment and prophylaxis clinical trials performed more than 25 years ago. Because of the availability of archived samples and clinical data from the Fred Hutchinson Cancer Research Center Long-Term Follow-Up program, we were able to establish a direct link between VL suppression and improvement in clinical outcomes at extended follow-up times. Because of the success of these studies, it is no longer ethical to include placebo arms in clinical trials because patients would progress to CMV disease at much higher rates than current standards of care allow. In the treatment trial and to a slightly lesser extent the prophylaxis trial, CMV disease occurred in a large percentage of patients, providing a clinical endpoint–rich environment and a dynamic range of CMV VLs that will not be observed in any future CMV treatment trials.

We evaluated VL kinetics, rather than VL itself, because whereas HIV and hepatitis C VLs respond to antiviral treatment with a stereotypic decline (28, 29), CMV VL response to treatment is more variable and depends somewhat on the immunological status of the transplant recipient (30, 31). In fact, we demonstrated that at some time points, including absolute lymphocyte count, an indicator of CMV immune recovery (27), improved many of the predictions of clinical outcomes from the Super Learner models. Moreover, spontaneous clearance of virus in the absence of treatment is also often observed (32). We included multiple CMV VL kinetics in our analysis to determine which aspects of this variability correlate with clinical outcomes (32–34). In the treatment trial, we found that mean and peak VL are valid surrogates of both CMV disease and the composite outcome of CMV disease or death. In the prophylaxis trial, we identified shedding rate as a surrogate of CMV disease. This difference between surrogate kinetics based on treatment setting may be significant in terms of the underlying biology of CMV under treatment versus prophylaxis. In the treatment setting, the magnitude of VL may be more predictive of tissue damage, whereas the number of viral reactivations under prophylaxis may portend a higher risk of CMV disease. Considering these differences may be important in designing future antiviral trials based on VL-based surrogate endpoints.

The main limitation of our study — that the data on which it is based have emerged from trials performed in an earlier era of HCT — is also its strength. In the treatment RCT, because patients were not treated with ganciclovir until viral cultures were positive, VLs as measured by CMV PCR at the time of randomization were much higher and more variable than current standards of care allow (35). It is precisely this large range of VLs that has allowed us to show which aspects of VL are most predictive of CMV-related clinical outcomes. This would not be possible using data from the modern era. The existence of placebo groups in these trials demonstrated that decreasing VL with an antiviral is the mechanism by which patients in the treatment group were protected from tissue-invasive disease.

In addition, applying our methods to VL data obtained from the prophylaxis RCT clarified that our findings are generalizable to lower VLs. Notably, the ganciclovir prophylaxis trial design bears some remarkable similarities with the recent letermovir phase III RCT. In both trials, antivirals were given early after transplant and continued through day 100 after HCT, and clinically significant infection and disease outcomes were assessed at 24 weeks, suggesting that our findings are relevant for modern clinical practice.

However, HCT practices have changed in several important ways since the historic ganciclovir trials were conducted. Both clinical trials we analyzed included only patients who received myeloablative conditioning prior to BMT, and the patient population was considerably younger than modern transplant recipients (36). In the current era, most patients receive peripheral blood stem cell transplants (PBSCTs) or umbilical cord transplants (UCTs) rather than BMT. In terms of CMV infection and disease risk, PBSCT recipients more often develop CMV infection and disease in the early posttransplant period (first 100 days) than in BMT, but rates are similar later after transplant when we assessed clinical outcomes (37). CMV infection occurs more frequently in recipients of UCT (38). In the Goodrich et al. studies, the majority of transplants were matched-related (68% in the treatment study, 52% in the prophylaxis study); a smaller percentage were unrelated (19% in the treatment study, 23% in the prophylaxis study). In the modern era, mismatched and haploidentical transplants are increasingly common (1, 2). Recipients of mismatched transplants have higher rates of CMV disease (39) and haploidentical transplant recipients have higher peak VLs (40). Modern HCT has increasingly employed alternative donors (41–43), and in these settings, recipients have higher rates of infection and disease and higher VLs, rendering our results relevant.

In addition, all Goodrich et al. study participants received myeloablative conditioning, whereas reduced-intensity conditioning regimens are now used frequently. Patients receiving reduced-intensity conditioning are less likely to have high-grade CMV infection (44), but overall rates of infection are similar (44, 45). Also, on average, patients in both the treatment and prophylaxis trials were in their early 30s, and the oldest patient in either trial was 56 years old. Whereas age has not been found to be a major risk factor for CMV reactivation or disease after HCT (39, 45), it is likely that age, cell source, donor match and relatedness, and conditioning regimens play a role in CMV-specific immune regulation after HCT, and we must acknowledge these limitations in our study data.

With those stated limitations and despite many changes in HCT practices, CMV disease and mortality continue to occur more frequently when VLs rise to higher levels in all risk groups (1, 2, 44), supporting the validity of our study in the modern HCT setting. Using data from placebo-controlled RCTs, we directly showed that reducing VL is the mechanism through which CMV disease is reduced—a mechanism that applies to treatment and prophylaxis and both early and modern settings.

In conclusion, this study provides evidence from 2 placebo-controlled RCTs, using state-of-the-art statistical and machine learning techniques, that CMV VL kinetics are valid surrogate markers for CMV disease or death in HCT recipients. These results strengthen the premise of current regulatory statements (46) and the recent clinical trials leading to approval of letermovir (9, 47). CMV VL kinetics could become powerful tools for developing and guiding the use of CMV drugs and immunotherapies for treatment or prophylaxis. In addition, our analytic approach could serve as a framework for validating surrogate markers for other viral infections, facilitating antiviral drug and immunotherapy development to eliminate viral complications after HCT.

Original study designs. Study methods for the original RCTs are included in the Supplemental Methods (17, 18). Briefly, in the early treatment study (17), CMV seropositive recipients or recipients of seropositive allogeneic BMT for hematologic malignancies at the Fred Hutchinson Cancer Research Center from 1989 to 1990 underwent weekly CMV surveillance with viral cultures from blood, urine, and throat swabs. If any surveillance cultures were positive prior to day 80 after transplant, patients were randomized to receive ganciclovir or a placebo, stratified by the presence of aGVHD, through day 100 after HCT. The primary endpoints were CMV disease (confirmed by biopsy or culture) and death by day 100 after transplant. Patients were observed for outcomes until day 180. CMV disease events were defined according to established guidelines (48).

In the prophylaxis study (18), CMV-seropositive recipients undergoing allogeneic BMT for hematologic malignancy requiring total body irradiation or busulfan-cyclophosphamide were randomized at marrow engraftment to receive ganciclovir or a placebo through day 100 after HCT or until a study endpoint of CMV infection (positive viral culture from surveillance culture), CMV disease, neutropenia, or death was reached. Additional clinical trial methods are available in the Supplemental Methods.

Extended clinical outcome analysis. We extended outcome assessment in both original RCT populations for CMV disease and mortality to 20 years by reviewing records maintained by the Fred Hutchinson Cancer Research Center Long-Term Follow-Up program. See additional details in the Supplemental Methods.

VL testing. Leftover clinical samples were stored in a repository from all patients undergoing HCT who gave their consent under a research protocol approved by the IRB. From this repository, we identified frozen serum samples spaced at approximately weekly intervals from day 0 to 100 after transplantation. The University of Washington Molecular Virology Laboratory performed quantitative CMV DNA PCR testing using a laboratory-developed assay (49). The assay’s limit of quantification is 71 IU/mL; the limit of detection is 36 IU/mL. Additional information about the assay is given in the Supplemental Methods.

VL kinetics. We determined baseline VLs at or near randomization and binned subsequent VLs into weekly intervals for 5 weeks after randomization. VL data collected after diagnosis of CMV disease were removed from analysis. VL was defined as the log₁₀-converted VL measured in IU/mL. Maximum change in VL was calculated by subtracting week 1 through week 5 VL from the baseline VL and finding the maximum of these values. Mean VL was defined as the average VL from week 1 through 5. Peak VL was defined as the highest VL measured from week 1 to 5. Percentage of positive VLs (shedding rate) was defined as the percentage of available weekly VLs above the limit of detection. Additional details regarding the timing of VL samples and calculation of VL kinetics are provided in the Supplemental Methods.

ALC kinetics. We determined ALC at randomization by choosing the ALC measured on the day of randomization or 1 day prior if the randomization day ALC was not available. The peak ALC was the highest ALC from randomization to 35 days (5 weeks) after randomization; ALC nadir was the lowest ALC from randomization to day 35 after randomization; mean ALC was the average ALC from randomization to day 35 after randomization.

Clinical endpoints. CMV disease (right-censored for death) and the first event of CMV disease or death were defined as the clinical outcomes of interest. We performed surrogate analyses on the occurrence of these endpoints by time from randomization/treatment initiation rather than time from transplantation. Thus, whereas the original study (17) defined clinical endpoints at 100 and 180 days after HCT, we defined clinical endpoints for the surrogate analysis at weeks 8, 24, and 48 after randomization (Figure 1C). For the early treatment trial, week 8 after randomization was chosen as the first clinical outcome to approximate the RCT’s study endpoint because all of the clinical endpoints that occurred by 100 days after transplant had occurred by week 8 after randomization. For the prophylaxis trial, patients were randomized earlier (at engraftment rather than positive viral culture), and thus, week 14 (approximately 100 days after randomization) was chosen. For both studies, 24 and 48 weeks were chosen as later endpoints to approximate 180 days and 1 year after randomization.

Statistics. All statistical analyses were performed in R (version 3.5.0) (50). Additional information regarding the methods, including all R packages, their versions, and Super Learner libraries used, are provided in the Supplemental Methods.

Survival and cumulative incidence analysis. Survival and first event of CMV disease or death curves were estimated using Kaplan-Meier methods. The cumulative incidence of CMV disease with death as a competing risk was estimated using the Aalen-Johnson method. Survival distributions and times to the composite endpoint of CMV disease or death were compared using the log-rank test. Cumulative incidence distributions for CMV disease with death as a competing risk were compared using Gray’s test. Throughout the analysis, differences were considered significant when P values were less than 0.05 unless otherwise indicated. All P values were 2-sided, and no adjustments were made for multiple hypothesis testing.

Validation of surrogate markers under the Prentice criteria. The Prentice criteria are met when a hypothesis test of the treatment effect on the surrogate endpoint is a valid hypothesis test of the treatment effect on the clinical endpoint (22). We evaluated whether each VL kinetic satisfied the first Prentice criterion by comparing the mean values of the VL kinetics in the ganciclovir and placebo groups using a 2-tailed Student’s t test. We evaluated the second Prentice criterion using logistic regression models of the association between each VL kinetic marker and each clinical endpoint, adjusting for baseline characteristics: aGVHD, donor CMV serostatus, and randomization VL but not treatment group. We evaluated the third Prentice criterion using logistic regression models of the association between each VL kinetic marker and each clinical endpoint, adjusting for treatment group and baseline characteristics: aGVHD, donor CMV serostatus, and randomization VL. The second Prentice criterion was satisfied if the coefficient of the VL kinetic term was significantly different from 0, indicating a significant association between the clinical endpoint and VL kinetic. The third Prentice criterion was satisfied if the coefficient of the treatment assignment term was close to 0 (P ≥ 0.20), i.e., when holding the value of the VL kinetic constant, the outcome was not more likely to occur in one of the treatment groups. In addition, if there was evidence of effect modification between a VL kinetic and treatment group (P < 0.20 in a logistic regression model containing an interaction term), the third criterion was not satisfied.

Percentage of treatment effect captured. We quantified how much of ganciclovir’s effect on clinical outcomes could be attributed to its effect on VL kinetics. This quantity is called the proportion of treatment effect captured by the surrogate (PCS) (23). A PCS higher than 63% is considered “moderate;” PCS higher than 85% is “substantial”; and PCS higher than 93% is “almost perfect” (23). Details regarding the PCS method and our implementation are provided in the Supplemental Methods.

Super Learner ensemble machine learning. Super Learner is an ensemble machine learning method that estimates a conditional outcome risk model as the optimal combination of individual regression algorithms that maximize a cross-validated criterion for best disease classification accuracy. Specifically, we minimized the leave-one-out, cross-validated area under the ROC curve (cv-AUC). Super Learner prediction models were built with the same baseline covariates and VL kinetics defined for the logistic regression analysis and were fit on data from the placebo group alone, the ganciclovir group alone, and the combined treatment groups, with individual regression algorithms specified in the Supplemental Methods. cv-AUCs were calculated for each Super Learner prediction model, with a predefined benchmark that cv-AUCs greater than 85% would provide evidence for the fitted values (i.e., predicted outcome risks) as potentially valid surrogates. Super Learning was implemented using the R package Super Learner (51).

Data sharing. Deidentified individual participant data may be requested from the corresponding author for further research.

Study approval. The original studies were approved by the Fred Hutchinson Cancer Research Center IRB and the FDA. All patients or their legal guardians provided written informed consent. The VL surrogate study was also approved by the Fred Hutchinson Cancer Research Center IRB.