Caring for patients in a new pandemic: the necessity and challenges of observational research

Individuals with coronavirus disease 2019 (COVID-19) can develop pneumonia and a severe inflammatory response with excessive cytokine release known as the cytokine storm. The JAK inhibitor baricitinib, used to treat rheumatoid arthritis, reduces inflammation by modifying the cytokine pathway. In this issue of the JCI, Bronte, Ugel, and colleagues performed an observational longitudinal study to evaluate the use of baricitinib in 20 patients with COVID-19. The treated patients showed reduced levels of plasma IL-6, TNF, IL-1β, and phosphorylated STAT3 as well as swift lymphocyte restoration. Notably, these patients had a dramatically favorable clinical outcome. While bias can plague uncontrolled research, this study has biological credibility and warrants randomized, controlled studies.

In 2020, severe acute respiratory syndrome–coronavirus 2 (SARS–CoV-2) took the world by storm and demanded immediate solutions from the biomedical research community. The first responses were a series of small studies that characterized the outcomes of patients who were given already-approved medications repurposed to treat coronavirus disease 2019 (COVID-19). A striking example is published in this issue of the JCI (1). On March 18, 2020, Bronte, Ugel, and colleagues (1) began studying the use of baricitinib in patients with COVID-19 pneumonia hospitalized at two hospitals in Italy. By April 18, they observed significantly lower mortality in patients given baricitinib, raising the question of whether baricitinib use itself reduced mortality (and should be given to others). The rapid accumulation of clinically important but uncontrolled COVID-19 data underscores the virtues and limitations of observational research.

Some of the most important discoveries in medicine have come from observational studies of groups of patients. For example, the links of tobacco use with lung cancer, HBV infection with hepatocellular cancer, and HPV with cervical cancer were all observed in patient cohorts. Of course, in these examples of disease causation, the alternative of a randomized, controlled trial is not possible. More relevant are the studies of HIV-infected persons that described a survival advantage for those on antiretroviral therapy at all stages of infection (CD4⁺ T cell counts), years before that benefit was proven by a randomized, controlled trial (2, 3). Thousands of lives were saved by adopting the practice prior to the proof coming from a randomized, controlled trial.

On the other hand, too often, uncontrolled findings are not confirmed, and with COVID-19, this concern is salient. Initial reports suggested that there were benefits of hydroxychloroquine with or without azithromycin, while others suggested harm or no effect (4, 5). Ultimately, randomized, controlled, double-blinded studies settled the matter (6). Lopinavir/ritonavir use suffered a similar fate (7, 8). The resulting misinformation was frustrating to providers, potentially harmful to patients, and reduced the public trust in medicine.

The Achilles heel of observational research is bias. Bias comes in many forms. The most intuitive is introduced by a difference in the patients who are treated. If the patients who are sicker are given a particular treatment, the benefits may appear diminished or be missed altogether. A similar masking of benefit can occur if the treatment is withheld as a “last hope,” but that severity of illness is not fully captured in the reference population. The opposite can also occur with a form of survivor bias. For example, if an acute infectious disease generally causes mortality in the first few days and it regularly takes three to five days to get a particular medication to the patient, then fewer of those who receive the medicine would die than historic controls — not because of the medicine but instead because it was differentially given to survivors.

There are many simple and some sophisticated epidemiological and biostatistical approaches to discounting bias and improving the validity of observational research. In principle, the groups compared are made as similar as possible in terms of factors likely to produce bias and/or made equivalent in their propensity to experience the outcome. Advances in these techniques have truly moved the field forward. However, they only work to the extent that all biases can be detected and measured. Important factors that are difficult to measure (such as why or when a physician gives a particular patient a medication) may notoriously still confound observational studies analyzed even with the best techniques.

The question of how to interpret observational research has become especially important during COVID-19. The rapid emergence of this lethal pandemic demands that clinical decisions be made in real time using all available data, which early in an epidemic mostly come from uncontrolled, observational research. The results of a randomized, controlled trial reported in September would not have helped Italian doctors on March 18. Biological plausibility is important and might have foretold that hydroxychloroquine was not going to work (and that ivermectin would not work at the doses we use in humans). However, although observational findings and biological plausibility supported the approach of blocking IL-6 signaling (9), on July 29, Roche announced that the first randomized, controlled trial of tocilizumab detected no benefit. Remdesivir and dexamethasone work, but, interestingly, chiefly in persons at different stages of COVID-19, and those findings were not widely anticipated (10, 11).

So what about baricitinib? Baricitinib inhibits JAK1 and JAK2 and is used to restrain the inflammation of rheumatoid arthritis. Since SARS–CoV-2 mortality involves a hyperinflammatory syndrome associated with elevated IL-6, which signals through JAK/STAT3, clinicians in Italy reasoned that JAK inhibition by baricitinib might improve the outcome of COVID-19 (12). A total of 20 persons were administered off-label baricitinib, starting with a loading dose to rapidly achieve steady-state. Only one patient died compared with 25 of the 56 other patients who were evaluated contemporaneously, and no baricitinib-related adverse events were reported (1). This important study raises the question of whether baricitinib is the next important breakthrough in the treatment of COVID-19 or the next example of the challenges of uncontrolled research.

There are several reasons to expect that the finding will be confirmed. First, the magnitude of the effect (difference in mortality) is enormous — only one of the 20 patients who received baricitinib died compared with 25 (45%) of 56 patients who were not treated. Very large effect differences are more likely to be confirmed (at least in principle if not scale) than are more subtle findings. Moreover, at the time of enrollment, the persons who received baricitinib were in some important respects similar to those who did not. Most notably, the pulmonary status (PaO₂/FiO₂ ratio) at time zero was similar in the two groups. The validity of the mortality findings is also supported by the detection of other expected associations, such as older age, male sex, and comorbidities. In addition, not only was there a priori biological plausibility that baricitinib might work, but also, in a subset of patients, the expected biological correlates of JAK inhibition were observed. Notably, compared with controls, in patients who received baricitinib, phosphorylation of STAT3 in T lymphocytes was inhibited; there was brisker restoration in lymphocyte populations; and plasma IL-6, TNF, and IL-1β levels dropped more rapidly. Presumably, the accelerated reduction of IL-6 levels was due to the blocking of a feed-forward loop (13). The findings of Bronte et al. (1) are also consistent with a few similar studies from others. Cantini et al. reported on 12 patients treated from March 16–30, 2020, and reported improved outcomes compared with contemporaneous controls (14). Tatanji et al. reported on 15 patients treated with baricitinib and hydroxychloroquine, 13 of whom experienced a quick resolution of their fever (15). However, some of the patients were at advanced stages of COVID-19 and/or underlying disease; three of the these patients ultimately died, and one had a pulmonary embolus (15).

There are also important reasons to be cautious when interpreting the Bronte et al. (1) data. The study is relatively small, despite the investigators (and medical providers) working under considerable adversity to enroll as many patients as possible. Although the investigators considered age, sex, and comorbidities, other differences in the groups are possible (and some are evident). Foremost, why did medical providers give baricitinib to some individuals but not others? Those treated had “a clinical onset of symptoms not exceeding nine days and the presence of interstitial lung involvement not exceeding 50% on chest x-ray or CT” (1). Logically, study participants without those attributes could have had more severe pulmonary disease or a longer duration of infection, although aside from the respiratory rate, no such differences were detected. Interestingly, those not treated with baricitinib died quickly. In fact, of the 56 patients in the control group, only 35 were still alive by day four; 37.5% is a very high four-day mortality rate, even for COVID-19, raising the question of what the baseline was for those not treated (1). If it took several days to identify and consent patients for the study and there was a sense it would work better for early-stage disease, the benefit of baricitinib might be smaller in another setting.

Thus, to the question of whether, on the basis of these findings (1), baricitinib should be routinely given to other patients, the answer is still no. These investigators are to be commended for achieving under extremely difficult circumstances what they set out to do with a 20-person nonrandomized trial — exploring the safety and justification of a definitive trial. However, in my view, in August 2020, baricitinib should only be used to treat COVID-19 in randomized, controlled trials. Fortunately, the United States National Institutes of Allergy and Infectious Diseases Adaptive COVID-19 Treatment Trial was designed for that very purpose: to compare baricitinib with placebo in patients taking remdesivir in a randomized, blinded fashion. In fact, by the time this article and Commentary are read, it is likely that the preliminary results of that study will be disclosed, and baricitinib will either be the third major advance in the treatment of COVID-19 or another example of the importance of trying all reasonable approaches to bring and end to this horrible pandemic (and an example of the limitations of uncontrolled research).

Conflict of interest: The author has declared that no conflict of interest exists.

Copyright: © 2020, American Society for Clinical Investigation.

Reference information: J Clin Invest. 2020;130(12):6225–6227. https://doi.org/10.1172/JCI143292.

See the related article at Baricitinib restrains the immune dysregulation in patients with severe COVID-19.

1. Bronte V, et al. Baricitinib restrains the immune dysregulation in patients with severe COVID-19. J Clin Invest. 2020;130(12):6409–6416.
   View this article via: JCI PubMed Google Scholar
2. Kitahata MM, et al. Effect of early versus deferred antiretroviral therapy for HIV on survival. N Engl J Med. 2009;360(18):1815–1826.
   View this article via: PubMed CrossRef Google Scholar
3. BLundgren JD, et al. Initiation of antiretroviral therapy in early asymptomatic HIV infection. N Engl J Med. 2015;373(9):795–807.
   View this article via: PubMed CrossRef Google Scholar
4. Gao J, Tian Z, Yang X. Breakthrough: chloroquine phosphate has shown apparent efficacy in treatment of COVID-19 associated pneumonia in clinical studies. Biosci Trends. 2020;14(1):72–73.
   View this article via: CrossRef Google Scholar
5. Sarma P, et al. Virological and clinical cure in COVID-19 patients treated with hydroxychloroquine: a systematic review meta-analysis. J Med Virol. 2020;92(7):776–785.
   View this article via: PubMed CrossRef Google Scholar
6. Cavalcanti AB, et al. Hydroxychloroquine with or without azithromycin in mild-to-moderate COVID-19 [published online July 23, 2020]. N Engl J Med.https://doi: 10.1056/NEJMoa2019014.
7. Cao B, et al. A trial of lopinavir-ritonavir in adults hospitalized with severe COVID-19. N Engl J Med. 2020;382(19):1787–1799.
   View this article via: PubMed CrossRef Google Scholar
8. Hung IF, et al. Triple combination of interferon beta-1b, lopinavir-ritonavir, ribavirin in the treatment of patients admitted to hospital with COVID-19: an open-label, randomised, phase 2 trial. Lancet. 2020;395(10238):1695–1704.
   View this article via: PubMed CrossRef Google Scholar
9. Xu X, et al. Effective treatment of severe COVID-19 patients with tocilizumab. Proc Natl Acad Sci U S A. 2020;117(20):10970–10975.
   View this article via: PubMed CrossRef Google Scholar
10. Beigel JH, et al. Remdesivir for the treatment of Covid-19 – preliminary report. Reply. N Engl J Med. 2020;383(10):994.
    View this article via: PubMed Google Scholar
11. RECOVERY Collaborative Group, et al. Dexamethasone in hospitalized patients with Covid-19 – preliminary report [published online July 17, 2020]. N Engl J Med. https://doi.org/10.1056/NEJMoa2021436.
12. Stebbing J, et al. Mechanism of baricitinib supports artificial intelligence-predicted testing in COVID-19 patients. EMBO Mol Med. 2020;12(8):e12697.
    View this article via: PubMed Google Scholar
13. Chang Q, et al. The IL-6/JAK/Stat3 feed-forward loop drives tumorigenesis metastasis. Neoplasia. 2013;15(7):848–862.
    View this article via: PubMed CrossRef Google Scholar
14. Cantini F, Niccoli L, Matarrese D, Nicastri E, Stobbione P, Goletti D. Baricitinib therapy in COVID-19: a pilot study on safety clinical impact. J Infect. 2020;81(2):318–356.
    View this article via: PubMed Google Scholar
15. Titanji BK, et al. Use of baricitinib in patients with moderate and severe COVID-19 [published online June 29, 2020]. Clin Infect Dis. 2020. https://10.1093/cid/ciaa879.