SciCheck Digest
Randomized controlled trials — the highest standard of evidence — have found that hydroxychloroquine isn’t beneficial in treating hospitalized COVID-19 patients. Yet social media posts are claiming the drug works, and conservative outlets have touted an unpublished, and much-criticized, observational study as evidence of the drug’s effectiveness.
Full Story
Researchers have studied whether the antimalarial drug hydroxychloroquine — pushed by then-President Donald Trump as early as March 2020 — is effective in treating COVID-19. But randomized controlled clinical trials, in which one group of patients is given the medication and another receives standard care, haven’t found a benefit in mortality or other clinical outcomes in hospitalized patients.
A randomized controlled trial is considered the “gold standard” in evaluating whether treatments are effective because it reduces bias by randomly assigning participants to treatment or standard care/placebo groups. Participants can be “blinded,” meaning they won’t know to which group they’re assigned. Researchers, who also can be blinded, can more confidently evaluate whether a treatment led to different effects.
Data from such a trial, among other evidence, led the Food and Drug Administration in June 2020 to revoke an emergency use authorization it had issued three months earlier giving some patients easier access to hydroxychloroquine and chloroquine if they couldn’t participate in a clinical trial. In revoking the EUA, the agency said the drugs, which are also used to treat lupus and rheumatoid arthritis, were “unlikely to be effective in treating COVID-19 for the authorized uses in the EUA.” The FDA said the “known and potential benefits” of the drugs “no longer outweigh” the risks.
But a year later, in a June 12 statement, Trump said he was “right” about several things, including: “Hydroxychloroquine works.”
No new randomized controlled trials have emerged to show that since we last looked at the research on the drug.
The only FDA-approved treatment for COVID-19 is the antiviral drug remdesivir, which the FDA approved in October for COVID-19 patients 12 years of age and older requiring hospitalization.
Under its Coronavirus Treatment Acceleration Program, the FDA says it has reviewed more than 460 trials and that hundreds more drug development programs are being planned. As of late June, 11 drugs and biological products have emergency use authorization from the FDA for COVID-19.
As we noted, the FDA revoked its EUA for hydroxychloroquine. But social media posts have promoted Trump’s claim that the drug “works.” Other posts claim hydroxychloroquine would’ve made the vaccines unnecessary, and others falsely say that emails from Dr. Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases, show he was secretly telling a family member to take the drug along with an antibiotic. As we’ve written, Fauci’s emails, made publicly available thanks to journalists’ Freedom of Information Act requests, don’t show that. One person who wrote to Fauci said that about hydroxychloroquine.
Conservative outlets, including One America News Network and Fox News, have recently publicized an observational study that hasn’t been peer-reviewed as having “confirmed hydroxychloroquine to be effective in the treatment of COVID-19,” as OAN put it. Fox News’ Laura Ingraham, in a June 1 segment, called it a “landmark study” and asked one of the authors whether 100,000 American lives could’ve been saved throughout the pandemic if “people had an open mind about hydroxychloroquine.”
The author, Dr. Stephen M. Smith, said: “I hate to say things like that. But, yes, I mean, the data — you know, I just try and look at the data.”
But in an interview with FactCheck.org, Smith backed away from such a figure. “I don’t know how many deaths could’ve been prevented,” he said, adding that he didn’t know he would be asked that question during the Fox News segment. “Again where we failed was to really look at what was out there and to analyze it in a really constructive way,” Smith said.
Experts who have studied hydroxychloroquine say Smith’s paper — which looked back at the treatment of 255 hospitalized patients requiring ventilation in March and April 2020 — has several flaws, particularly with the way the statistical analysis was done. And again, they point to the stronger evidence from randomized controlled trials.
“Retrospective observational studies such as this cannot compare with the much larger randomized controlled trials which show unequivocally that HCQ is NOT beneficial in the treatment of hospitalised patients,” Nicholas White, a professor of tropical medicine at the University of Oxford’s Centre for Tropical Medicine and Global Health and Mahidol University in Thailand, told us in an email.
White is helping to run a multinational randomized controlled trial on whether hydroxychloroquine or chloroquine is effective as a preventive to COVID-19. His trial is continuing, he said, adding that the question on whether the drugs can provide a benefit for early treatment or prevention “remains open.”
Duke University’s Clinical Research Institute is leading another randomized clinical trial on whether hydroxychloroquine can prevent COVID-19 among health care workers. The first two randomized controlled trials to announce their findings on hydroxychloroquine as a post-exposure prophylaxis did not find a benefit, including one trial from infectious disease specialist Dr. David Boulware and colleagues at the University of Minnesota.
Boulware was also critical of Smith’s unpublished observational study. “This statistical analysis is completely invalid and therefore the conclusions are invalid,” he told us.
The observational study was posted on a preprint website on May 31. A preprint hasn’t been peer-reviewed, and the site says that while posting these unpublished papers can allow scientists to see and comment on findings, it also means the papers “have not been finalized by authors, might contain errors, and report information that has not yet been accepted or endorsed in any way by the scientific or medical community.”
The study was authored by researchers at the Smith Center for Infectious Diseases & Urban Health, founded by Smith, and Saint Barnabas Medical Center in New Jersey. In the past, Smith has been a proponent of hydroxychloroquine, telling Fox News in April 2020 that it was “a game-changer” and that he had been using it, along with the antibiotic azithromycin, to treat 72 COVID-19 patients.
Dr. Neil Schluger, the Barbara and William Rosenthal Chair of the Department of Medicine at New York Medical College and director of medicine at Westchester Medical Center, told us the study “wouldn’t get a passing grade in a beginning course in statistics in any school of public health in the United States,” and that he didn’t think it would survive peer review.
Schluger co-authored a peer-reviewed observational study — published in the New England Journal of Medicine in May 2020 — on COVID-19 patients admitted to a hospital in New York. It found that “hydroxychloroquine administration was not associated with either a greatly lowered or an increased risk” of death or intubation. It said randomized controlled trials of the drug were needed, and Schluger told us his study has since been superseded by such trials.
Smith defended his study against the criticisms. He also said it was “simplistic” for some to claim that his study showed hydroxychloroquine alone was effective, saying that “the data don’t show” the drug “works.” Rather, he said it was the combination of hydroxychloroquine with the antibiotic azithromycin — with both drugs administered above a certain level — that was associated with “a big increase in survival” in patients requiring invasive mechanical ventilation.
He told us that because his study looked at the combination of the drugs, there were no randomized controlled studies with which to compare his observational study, except for a randomized controlled study from Brazil. He had several criticisms of that study.
Two peer-reviewed, randomized controlled studies from Brazil looked at the impact of using hydroxychloroquine and azithromycin: One study focused on mild to moderate COVID-19 hospitalized patients, and the other enrolled patients with severe COVID-19. Both found the combination of the drugs didn’t improve clinical outcomes.
“No trial is perfect,” Schluger told us, “but an RCT [randomized controlled trial] trumps a small retrospective study just about any day of the week.”
What Studies Have Found
Before we get to the recent preprint, let’s review what those gold standard randomized controlled trials have found regarding hydroxychloroquine.
An article published by BMJ Journals explains the significance of randomization this way: “The main purpose of random assignment is to prevent selection bias by distributing the characteristics of patients that may influence the outcome randomly between the groups, so that any difference in outcome can be explained only by the treatment.”
In early June 2020, the chief investigators of the Recovery trial in the U.K. announced they had found no significant difference in mortality at 28 days and no evidence of benefits on other outcomes in using hydroxychloroquine. The study included 1,542 patients hospitalized in the U.K. with COVID-19 who were given hydroxychloroquine and 3,132 given usual care.
Results from the study were published in the peer-reviewed New England Journal of Medicine in early October. In the hydroxychloroquine group, 27% of the patients died within 28 days, and 25% of those in the usual-care group died.
Two other randomized clinical trials told a similar story.
The hydroxychloroquine arm of the Solidarity trial, an international study launched by the World Health Organization and partners to test the effectiveness of several possible COVID-19 treatments, was halted in mid-June due to data from that trial, the evidence from the Recovery trial and a review of other evidence, the WHO said.
Results from the trial were published in the New England Journal of Medicine in December. The study, which included 954 patients assigned to receive hydroxychloroquine, found that the drug, as well as some other treatments “had little or no effect on hospitalized patients with Covid-19, as indicated by overall mortality, initiation of ventilation, and duration of hospital stay.”
And the National Institutes of Health also studied the drug, halting its study in hospitalized patients in June 2020, after a data and safety monitoring board reviewed interim data. Peer-reviewed results were published in JAMA in November. The study said hydroxychloroquine “did not significantly improve clinical status at day 14″ among hospitalized patients with respiratory illness from COVID-19. “At 28 days after randomization, 25 of 241 patients (10.4%) in the hydroxychloroquine group and 25 of 236 (10.6%) in the placebo group had died.”
More recently, meta analyses have provided a look at the combined findings of many studies. One is a “living systematic review” of randomized controlled trials of several drug treatments for COVID-19 that continues to be updated when new findings emerge. Published by the British Medical Association’s BMJ, that meta analysis concluded: “Azithromycin, hydroxychloroquine, lopinavir-ritonavir, and interferon-beta do not appear to reduce risk of death or have an effect on any other patient-important outcome.” That’s according to the fourth version, last updated April 6.
Another meta analysis, published June 15 in Pathogens and Global Health, looked at 25 observational studies involving 41,339 patients and 11 randomized clinical trials involving 8,709 patients. It found hydroxychloroquine wasn’t associated with mortality in the pooled random clinical trials, which it called a “high level of certainty of evidence,” but the drug was associated with a 20% reduction in mortality in the combined observational studies, which it called a “low level of certainty of evidence.”
The meta analysis said the association with reduced mortality “was mainly apparent by pooling observational studies using lower doses of HCQ” and that the findings might “encourage” randomized controlled trials using low doses of the drug. But the analysis said findings from observational studies “should be considered with caution because the overall strength of evidence grade was judged to be low.”
As for using azithromycin along with hydroxychloroquine, the analysis said it “seems neither to increase nor decrease the effect, if any, of the HCQ,” but said this was “inconclusive because of the very large uncertainty in the findings.”
The randomized controlled trials that evaluated the combination of the drugs in Brazil produced two peer-reviewed studies. The first, published in the NEJM in July 2020, randomized patients hospitalized with mild to moderate suspected or confirmed COVID-19 in 55 hospitals to three groups: 217 patients received hydroxychloroquine and azithromycin, 221 received hydroxychloroquine, and 229 received standard care. Of those 667 patients, 504 patients had confirmed Covid-19. And among those with confirmed COVID-19, the study did not find that hydroxychloroquine, either alone or in combination with azithromycin, improved clinical outcomes compared with the standard care.
“In this open-label, multicenter, randomized, controlled trial involving hospitalized patients with confirmed mild-to-moderate Covid-19, a 7-day course of hydroxychloroquine either with azithromycin or alone did not result in better clinical outcomes as measured by a seven-level ordinal scale at 15 days,” the authors wrote. “There was also no effect on any of the secondary outcomes,” which included clinical status at seven days.
Smith had several criticisms of that study and reasons why it can’t be compared to his, including that some patients initially randomized weren’t confirmed to have COVID-19, that 36% of the participants had used azithromycin before enrollment, that the study didn’t concern very sick patients and that few died in each of the three groups, making it difficult for the hydroxychloroquine groups to do better on mortality.
The study notes the previous azithromycin use (and 9.3% of participants’ previous use of hydroxychloroquine) as a limitation. The authors noted that “in most instances, the duration of previous use was only 24 to 48 hours before enrollment,” because the study required patients to be enrolled within 48 hours of admission to the hospital for the bulk of the enrollment period and use of the drugs as an outpatient was “infrequent.” For the remainder of the enrollment period, patients who had used the drugs for more than 24 hours were excluded.
Smith is right that few died – five in the group with both drugs, seven in the hydroxychloroquine group and six in the control. But the study was on mild to moderate COVID-19 cases, and it measured other health outcomes besides mortality.
Researchers in Brazil published a second study — on those with severe COVID-19 — in the Lancet in September. That randomized clinical trial found that giving patients with severe disease azithromycin along with hydroxychloroquine, as opposed to hydroxychloroquine alone, “did not result in clinical improvement or mortality reduction.”
At the time the study was conducted — from March 28 to May 19, 2020, 447 patients were enrolled — Brazilian health authorities recommended the use of hydroxychloroquine as standard care for patients with severe COVID-19, so the “standard care” group was given hydroxychloroquine. It wasn’t possible at the time to have a group of patients not using that drug.
The Recent Observational Study
The unpublished observational study posted online in late May looked at 255 COVID-19 patients at Saint Barnabas who required invasive mechanical ventilation. Nearly all had been admitted to the medical center from March 12, 2020, to May 1, 2020. (One contracted COVID-19 at the hospital.)
The vast majority of patients — 87.8% — were given hydroxychloroquine, and 62.5% of those also got azithromycin. (Some patients also received other treatments that the study analyzed: corticosteroids; tocilizumab, a monoclonal antibody that targets inflammation; and convalescent plasma.) The overall survival rate was poor: 78.8% of them died. They were observed for up to 90 days of hospitalization or the time they were discharged. Among the 54 who survived, only nine were discharged to their homes “without any cognitive or motor deficits and off oxygen therapy.” (But the study notes the outcome for seven in the group of 54 survivors is unknown.)
The study concluded that modeling “established higher doses of HCQ and AZM greatly improved survival” in COVID-19 patients who required invasive mechanical ventilation and that administering the drugs adjusted by patient weight “improves survival by over 100%.”
It said 18 out of 37 patients who got a higher cumulative amount of hydroxychloroquine and azithromycin survived, compared with 36 surviving patients among the 218 who did not receive the higher cumulative amounts. From those figures, the study calculates a nearly 200% relative difference in survival.
“Differences of these magnitudes have not been reported in other clinical studies,” the authors wrote.
Smith told us he thought there would be some benefit to using the drugs, before the analysis was done, but he “didn’t know if the result would be that dramatic.” He called the data on the combination of the two drugs “stunning,” and he said if the mortality difference had been a good bit lower, “people would be more willing to listen.”
But the experts we interviewed told us the data weren’t analyzed properly.
Schluger told us the study most likely found the association between higher cumulative doses and survival because “in order to receive a higher cumulative dose you had to be alive.” The problem is called an immortal time bias. Patients weren’t randomized to get higher doses or not; instead it was done at the doctors’ discretion, Schluger said.
Boulware, too, told us the study suffered from an immortal time bias, which he called a “basic statistical concept.” The analysis compares “people who survived 10 days to people who did not survive 10 days,” he said in an interview. “Not surprisingly the people who survive 10 days do better.”
The groups the study compares “have to be defined at baseline, otherwise it’s a bias,” he said in a series of tweets on the preprint.
In terms of drawing conclusions on hydroxychloroquine, the study “doesn’t compare getting hydroxychloroquine with not getting it,” Boulware said. The vast majority of all patients got the drug.
“This will not get published,” Boulware said. “Any legitimate journal will recognize the problem with the analysis.”
Smith objected to the immortal time bias criticism, emailing us a list of several reasons he felt the study didn’t have such a bias. He said that a “higher dose of HCQ alone was not associated with an increase in survival,” and that patients would pass the higher cumulative dose thresholds – 3 grams of hydroxychloroquine and 1 gram of azithromycin — by day five of hospitalization, if they were prescribed that and started on day one. He said “the vast majority of patients was alive long enough” to have received the higher cumulative doses, again if they were prescribed it.
He also cited weight-based cumulative hydroxychloroquine dosing, saying it “was more strongly associated with survival than absolute cumulative HCQ dose,” and that younger patients weigh more.
“When one uses weight-based cumulative HCQ dose, younger [patients] move down the relative scale and older [patients] up. Weight-based dosing, however, removes or drastically reduces the possibility of immortal time bias, meaning it removes the chances that HCQ cumulative dose is a marker of survival, not causally linked.”
But Schluger and Boulware disagreed.
“In describing ‘cumulative dosing,’ that is the essence of immortal time bias,” Boulware said in an email. Adjusting for weight doesn’t fix that, he said. “Weight is a risk factor for COVID-related mortality, so by adjusting for weight based dosing, one is adjusting for the underlying risk of mortality due to weight. The proper analysis remains a time-dependent Cox-regression model. One could include weight as a covariate in the model. They did not do this (but could).”
“Dr. Smith’s study is certainly an example of immortal time bias, as many people have pointed out,” Schluger said. “There is no indication anywhere in the paper that hydroxychloroquine was administered by weight-based dosing. It wasn’t. It turned out that patients who received a total of 2-3 grams of HCQ had received a higher cumulative mg/kg dose, but that is because they were alive long enough to have gotten that much. It wasn’t as if they received that total, higher cumulative weight-based dose intentionally on day 1 or 2 of hospitalization, and that’s why this is clearly an example of immortal time bias.”
None of the statisticians who ran the data through a causal model is named as an author on the study, Smith said. He told us he had offered co-authorship to the statisticians, but that higher-ups in the firm didn’t want to be associated with hydroxychloroquine.
Overall, Schluger said, “The study is a small, retrospective one, and any definitive claims based on its findings need to be taken with a very large grain of salt.”
Schluger said in an email that there were “several obvious differences that are striking” between the patients who survived and those who died, including that the latter were 13 years older on average. “That’s a huge difference, and age has always been the strongest risk factor for mortality.”
Smith told us that he would make the database for the study public. And he told us the paper had been submitted to the journal PLOS ONE and is going through the peer-review process. He said it had been more than four weeks (and by now, it would be more than five weeks). “I’ve never seen a gap like this,” he said of the time it was taking, and he said the journal is consulting a fourth reviewer, which he said was unusual.
PLOS ONE wouldn’t confirm whether the preprint had been submitted to the journal, citing its confidentiality policy. David Knutson, senior manager of communications for the journal, told us the average time from submission to publication is about 160 days, a process that involves “multiple rounds of peer review,” revisions by the author and feedback from reviewers. “Most PLOS ONE articles receive input from two referees and the handling editor, although the number of reviewers may vary depending on the article and the handling editor’s own expertise,” he said.
Editor’s note: SciCheck’s COVID-19/Vaccination Project is made possible by a grant from the Robert Wood Johnson Foundation. The foundation has no control over our editorial decisions, and the views expressed in our articles do not necessarily reflect the views of the foundation. The goal of the project is to increase exposure to accurate information about COVID-19 and vaccines, while decreasing the impact of misinformation.
Sources
Food and Drug Administration. Letter revoking EUA for chloroquine phosphate (CQ) and hydroxychloroquine sulfate (HCQ). FDA.gov. 15 Jun 2020.
Food and Drug Administration. Letter of authorization for chloroquine phosphate and hydroxychloroquine sulfate. FDA.gov. 28 Mar 2020.
Statement by Donald J. Trump, 45th President of the United States of America. donaldjtrump.com. 12 Jun 2021.
McDonald, Jessica. “Navarro Doesn’t Give Full Picture On Hydroxychloroquine.” FactCheck.org. 9 Jul 2020.
Food and Drug Administration. Coronavirus Treatment Acceleration Program (CTAP) webpage. FDA.gov. current as of 9 Jun 2021.
Food and Drug Administration. FDA Approves First Treatment for COVID-19. FDA.gov. 22 Oct 2020.
Food and Drug Administration. Emergency Use Authorization webpage. FDA.gov. current as of 30 Jun 2021.
Spencer, Saranac Hale et. al. “Viral Posts, Pundits Distort Fauci Emails.” FactCheck.org. 4 Jun 2021.
“The Ingraham Angle.” Fox News. 1 Jun 2021.
Smith, Leon G. et. al. “Observational Study on 255 Mechanically Ventilated Covid Patients at the Beginning of the USA Pandemic.” medrxiv.org. 31 May 2021.
Smith, Stephen M., executive director and founder, Smith Center for Infectious Diseases and Urban Health. Phone interview and email to FactCheck.org. 24 Jun 2021.
White, Nicholas, professor of tropical medicine at the University of Oxford’s Centre for Tropical Medicine and Global Health and Mahidol University in Thailand. Email to FactCheck.org. 17 Jun 2021.
Duke Clinical Research Institute. Duke Clinical Research Institute to Expedite Study of COVID-19 Prevention in Healthcare Workers. press release. 1 Apr 2020.
Mitjà, Oriol et. al. “A Cluster-Randomized Trial of Hydroxychloroquine for Prevention of Covid-19.” New England Journal of Medicine. 384.5 (2021).
Boulware, David R. et. al. “A Randomized Trial of Hydroxychloroquine as Postexposure Prophylaxis for Covid-19.” New England Journal of Medicine. 383.6 (2020).
Boulware, David, infectious disease specialist, University of Minnesota. Phone interview and emails to FactCheck.org. 16 and 24 Jun 2021.
“The Ingraham Angle“. Fox News. 1 Apr 2020.
Schluger, Neil, the Barbara and William Rosenthal Chair of the Department of Medicine at New York Medical College and director of medicine at Westchester Medical Center. Phone interview and email to FactCheck.org. 17 and 24 Jun 2021.
Geleris, Joshua et. al. “Observational Study of Hydroxychloroquine in Hospitalized Patients with Covid-19.” New England Journal of Medicine. 382.25 (2020).
Cavalcanti, Alexandre B. et. al. “Hydroxychloroquine with or without Azithromycin in Mild-to-Moderate Covid-19.” New England Journal of Medicine. 383.21 (2020).
Furtado, Remo H M et. al. “Azithromycin in addition to standard of care versus standard of care alone in the treatment of patients admitted to the hospital with severe COVID-19 in Brazil (COALITION II): a randomised clinical trial.” The Lancet. 396.10256 (2020).
Akobeng, A K. “Understanding randomised controlled trials.” Archives of Disease in Childhood. 90.8 (2005).
The RECOVERY Collaborative Group. “Effect of Hydroxychloroquine in Hospitalized Patients with Covid-19.” New England Journal of Medicine 383.21 (2020).
WHO Solidarity Trial Consortium. “Repurposed Antiviral Drugs for Covid-19 — Interim WHO Solidarity Trial Results.” New England Journal of Medicine 384.6 (2021).
Self, Wesley H. “Effect of Hydroxychloroquine on Clinical Status at 14 Days in Hospitalized Patients With COVID-19: A Randomized Clinical Trial.” JAMA. 324.21 (2020).
Siemieniuk, Reed AC et. al. “Drug treatments for covid-19: living systematic review and network meta-analysis.” BMJ 2020. Fourth version updated 6 Apr 2021.
Di Castelnuovo, Augusto et. al. “Hydroxychloroquine and mortality in COVID-19 patients: a systematic review and a meta-analysis of observational studies and randomized controlled trials.” Pathogens and Global Health. 15 Jun 2021.
Knutson, David, senior manager of communications, PLOS. Email to FactCheck.org. 30 Jun 2021.