Pandemic science is a rollercoaster. Almost every week, there’s a new first-rate piece of research that helps make serious progress in the fight against Covid-19. At the same time, there are usually clear examples of how not to do science. Last week was no exception.
Our dose of good science came in the form of the immunosuppressive drug tocilizumab. The Prime Minister had some trouble pronouncing its name at a press conference, and I don’t blame him. But however you say it, Oxford’s Recovery trial has found it seems to work.
The new study hasn’t yet been peer-reviewed – it’s been posted online as a preprint, the preliminary kind of paper released for discussion prior to (or during) review. It reports a relatively small but still important effect: a 14 per cent reduction in the risk of death from Covid-19 for patients who were given tocilizumab (along with the known benefit of steroids, which most of the patients in the trial were also taking). Based on this finding and a previous, smaller study, tocilizumab is now available to coronavirus patients on the NHS.
That’s the good news. Here’s the bad: the website Retraction Watch last week announced that its list of retracted coronavirus-related scientific papers has now added its 84th entry. The paper in question was on Covid and obesity, and was withdrawn from the scientific literature due to:
“inadvertent errors… that unfortunately passed unnoticed during the extremely rapid review and publication process at the peak of the Covid-19 pandemic”.
Other papers on the Retraction Watch list have been pulled due to dodgy data, dodgy peer-review and plagiarism; some were retracted for unknown reasons.
So why am I so optimistic about the tocilizumab study? Why am I betting it won’t go the same way as the inglorious 84? What should we be looking for that distinguishes good and bad research on Covid – or, indeed, on anything else?
The first point is transparency. The Recovery team has been admirably open about how its data would be analysed. Although every clinical trial has to be registered, it’s still depressingly common for researchers to be overly flexible: changing the measurements they’re focusing on, shifting to a different type of statistical test if they get disappointing results, or splitting the data up into subgroups on a whim. All this makes results less reliable, since the more analyses you run, the more likely your finding is to be a fluke.
But the Recovery trial had an analysis plan, written and posted online before anyone saw the data. The researchers stuck to it, and whenever they wanted to run an analysis they hadn’t planned, they simply said so (several times in the preprint, they note they ran an “exploratory” analysis, and the reader can thus take the results with a pinch of salt). Being open about the often messy process of doing research is refreshing in a world where most scientific papers are written as if everything were planned in advance, but where that’s hardly ever true.
I mentioned that tocilizumab has a relatively small effect – as the researchers put it, on average you have to treat 25 patients with the drug to prevent one death. But that small effect is, perhaps paradoxically, our second reason to be optimistic. Very large effects aren’t particularly common in medical treatments (for drugs, at least… let’s leave aside the obvious large effect that “surgery to remove a bullet lodged near an artery” would have). If the effect had been apparently dramatic, it would have raised suspicions, since such outcomes are usually too good to be true.
That kind of suspicion is currently being levelled at a new study of Vitamin D for Covid-19, also recently released as a preprint. I’ve previously written about the ambiguous evidence suggesting Vitamin D can help combat the virus, but this preprint appeared to be the clincher: a randomised controlled trial showing, as the MP David Davis explained in a viral tweet, an 80 per cent reduction in intensive care visits and a 60 per cent reduction in deaths.
Sadly, the study was not, in fact, a randomised controlled trial (the authors have now acknowledged this, despite what they previously wrote in their preprint). There are also several problems with its methodology. But the main results themselves should have had us on guard: it’s a priori implausible that a common vitamin would cut Covid-19 deaths by well over half.
That doesn’t rule out a smaller effect of Vitamin D, and we might find one as future trials – trials that are actually randomised and controlled – report back. But a 60 per cent reduction in deaths? Come on, man.
Its transparency and its plausibility are just two reasons to feel positive about the Recovery trial. There are other reasons, such as its large sample size, its solid randomised design and the good, theoretical grounds to expect an anti-inflammatory drug like tocilizumab to help with a disease such as Covid-19. You won’t find many of these advantages in the now-retracted studies or in the Vitamin D “trial”.
But let’s look on the bright side. The contrast between the highs and lows of Covid-19 research is a good lesson in differentiating solid from shoddy science.
[see also: Does vitamin D really help combat Covid-19?]
Content from our partners
Related
