"OMFG", cried the collective headlines of the worlds' news-starved press the other day, "there's a vaccine for AIDS!" This of course would be awesome news for people who enjoy sex (and terrible news for people who think sex is the Devil's work and should be punished by disease and pestilence) if it were true. But as soon as you start digging behind the headlines, the claims start to look very ropey indeed.
Let's start with the reported facts, from the BBC. They relay the claims from the 16,000 person US/Thai trial run in Thailand as follows:
Researchers found that it reduced by nearly a third the risk of contracting HIV, the virus that leads to Aids. It has been hailed as a significant, scientific breakthrough, but a global vaccine is still some way off.
Now, reducing AIDS by a third in vaccinated individuals sounds like quite an impressive result, but let's look at the actual numbers here.
74 [out of 8198] people who did not get the vaccine infected and 51 [out of 8197] of the vaccinated group infected.
Now, these are very small numbers, and testing them for stastical significance gives some interesting results. For a result to be statistically significant, it needs to have a p-value of less than 0.05. That means that there is less than a 5% chance of the result occuring purely by chance. In the case of these results, I calculate that p = 0.048, which is pretty much right on the boundary.
(Edit: DMcIlroy gets the same results here)
So the results then were statistically significant... barely. In fact, if just one or two people had gone the other way, the results would have fallen below the level of statistical significance - that's how precarious the findings are.
That on its own should be enough to make people a bit cautious, but in combination with some other information coming out of this trial, alarm bells should be ringing. The BBC note that:
The vaccine was a combination of two older vaccines that on their own had not cut infection rates.
Pharmaceutical company Sanofi developed one of the two vaccines combined for this study, and even they admitted to the NYT that they were surprised by the results:
"This came out of the blue," said Chris Viehbacher, Sanofi’s chief executive. Even 31 percent protection "was at least twice as good as our own internal experts were predicting," he added.
To put this surprise result into further context, back in 2004 an array of scientists in the United States lined up to write a joint letter to Science magazine in which they heavily criticised the decision to fund the $100m+ trial, describing the scientific rationale as "weak" and pointing out that the components of the new vaccine had been shown to be either ineffective or poorly immunogenic.
So not only do we have a veritable symphony of surprised voices, but there are serious anomalies in the trial results themselves, as reported by the New York Times:
"...those who became infected had as much virus in their blood whether they got the vaccine or a placebo — suggests that RV 144 does not produce neutralizing antibodies, as most vaccines do, Dr. Fauci said."
There are several possible explanations for this unexpected result. It could be consistent, for example, with the vaccine working through a different pathway in the immune system, perhaps stimulating effector cells as Dr. Fauci of NIAID speculates in the NYT article. However, it would also be consistent with the vaccine simply not working, and the cold facts are that there is no evidence for an alternative hypothesis, certainly not without going back and studying the collected blood samples very closely.
So to summarize, the newspapers are suggesting we have a vaccine that is 30% effective. The scientists involved to their credit seem to be playing things down a bit, but frankly their results are tenuous.
We have a result that is barely statistically significant, using a vaccine comprised of two components previously shown not to work, a methodology heavily criticised by a galaxy of experts in a joint letter to Science, and quirks in the results which would be consistent with the vaccine not working.
I'm not going to stick my neck out and call this trial a failure, but no self-respecting skeptic can look at this results are declare them to be anything other than tenuous. A lot more evidence needs to be presented before we can take these results too seriously.
- - - - - - - - - - - - - - - - - - - - - - - - - -
Follow me on Twitter! @mjrobbins
http://layscience.net/trackback/640
You big evil poo-poo head, I was getting all excited. You do have some annoyingly good points though. Damn. Well I guess we'll have to wait for the study to be published to take a closer look
Sorry to disappoint! I agree about waiting for the trial to be published; in fact it's rather annoying that the paper hasn't already been published.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter
Have a look at what Lib Dem shadow science minister Evan Harris had to say about this - remembering that he was the first to be injected in a previous AIDS vaccine trial in 2000.
Sensible comments, particularly about waiting for publication before jumping for joy.
I'm a fan of Evan Harris, he's a very sharp bloke. Supporter of Simon Singh too.
Martin is the editor of layscience.net.
Follow Me!
RSS | Twitter
Amber flags went up for me on this one almost immediately.
a)Sub-editors headline not matching storyline
b) no mention of statistical signifcance
c) no mention of publication in any journal
... but I was too lazy to do the maths.
I'm with you on this one. Furthermore, it very much highlights the problem of choosing an arbitrary cut-off point at which results are dichotomised into "significant" and "non-significant". The p-value of 0.048 indicates that there would be a 4.8% chance of getting these (or more extreme) results, in a perfectly designed trial with no biases. Now, if we'd seen a p-value of 0.0048 then we might start to get excited, and think that the extreme low likelihood of the results being due to chance mitigates against some of the faults of the trial design.
I'm in Morocco at the moment, and saw some news coverage of the trial on French television. They interviewed one of the participants and immediately some alarm bells began to ring. He had been recruited onto the trial, appeared to have known that he'd been allocated to the experimental group, and then was encouraged to go recruit as many of his friends in the village where he lived to do the same. I would be extremely sceptical of the published results if they then have the tenacity to declare that this was a "double-blind randomised study", when it appears that this is not the case.
I can't find the news clip where they interviewed this trial participant - maddeningly I've found a clip where they show him pretend-giving out leaflets (0:51 on http://www.youtube.com/watch?v=NnaGQ6T-ack) but they've cut what he has to say. If anyone finds a clip of this I'd be grateful.
For further reading on the problem of dichotomising results into "significant" and "non-significant", please see Sterne and Davey Smith (2001), Sifting the evidence---what's wrong with significance tests? http://physicaltherapyjournal.net/cgi/content/full/81/8/1464.
Really good post, I'm glad you've managed to shed some light on the statistics. Just a question, was this trial repeated?
I guess we all have to wait for the article to be published.
I do wonder, will someone please elighten me, why was the US army involved in sponsoring this trial.
confused.com
Hardly a bastion of forward thinking scientific medical research?
Good post. My feeling is that this is a chance finding, although I'm no expert. But the thing about the "vaccine" not affecting viral load in the infected people clinched it for me, because that's exactly what you would expect if it were a chance result.
Overall, it comes down to a choice between believing a p=0.048 result could occur by chance, vs believing that something weird is going on immunologically.
So unfortunately I'm skeptical. But still, it'll be interesting to see what happens.
Michael: For more anti-p=0.05 stuff see here, there's even a whole book on it.
I think the statistical significance is even less than it would seem. Realistically, this is really the 3rd test of this vaccine, the first two tests having shown no positive effect. The proper null hypothesis seems like it should take into account those previous failed trails, on the assumption that eventually you'll find a 'significant' result if you just keep repeating the trial.
My quick tests by simulation say that there is really more like a 10% chance that one would find a false 'significant' result of this magnitude with 3 trials. Now, a 90% chance of a true result is nothing to scoff at, but I'd definitely want to look very closely for holes in the rest of the study before proclaiming that a vaccine has been found.
@Neuroskeptic:
Thanks for the information. On reading the reviews of McLoskey and Ziliak's book, it sounds rather vitriolic, judgemental and (although well-intentioned) rather unhelpful. I think I'll wait until Jonathan Sterne brings something out before I buy a book on the subject.
That said, I think the point to take away from all of these three authors is that the p-value represents a range of strength of evidence, rather than something which magically sets off fanfares and fireworks when it drops below a particular value. The p-value indicates moderate evidence; the fact that it has taken over 30 attempts (according to today's TV reports) to get a borderline result is cause for calm scepticism not alleluias. Furthermore, we're talking about incidence rates of less than 1 per thousand here, in both the experimental and control arms of the trial.
I hope, and indeed pray (sorry Martin), that false hopes are not raised here.
felt so moved to blog about it myself: http://www.nontoxic.org.uk/?p=128
You are right to be skeptical - the NNT from this trial was 357 (with an eye watering CI of 183 - 7032).
As it happens I posted on the same subject yesterday as well - see Sex and Stats. I'd been waiting for a news story to show how misleading relative risk reduction can be and this vaccine trial fitted the bill perfectly.
PS It seems we both use Drupal with a Marinelli (in my case somewhat customised) theme. Great minds think alike - or should that be Fools seldom differ?
@Dr. No
I have added your figures to my blog post. Thanks for the analysis.
By the way I emailed Katie Stover (Media Representative for NIAID) who said that it would be published in NEJM in October, to coincide with the AIDS Vaccine Conference
Some thoughts:
In trials of this nature, subjects are counselled about HIV and given appropriate advice and means on how to avoid infection. It would be quite unethical not to. This may paradoxically interfere with the chances of the trial revealing any early benefits, and you need many more people to be recruited. So the NNT (or NNV really) of 357 may be an overestimate of what this vaccine might do outside of the trial scenario in the field, as it were.
Is 357 too big a number to bother with vaccination? For something like HIV I don't believe so. Were it to be the NNT to avoid catching a cold, then they'd be laughed into touch.
Having said that, I can only endorse eveything that people have said here so far about the apparently modest effect of this vaccine. It will be interesting to see the authors explain away these concerns in the actual paper.
Good point, DT. That of course is why we differentiate between efficacy (what happens is trials when every one is doing their best) and effectiveness ( ie what happens in the real world).
I also agree the NNV (!) depends on the nastiness of the infection involved. But we don't really have enough data yet - remember that whopping CI.
Dr No
It just does not seem to me that there is enough people who will be protected. It seems as if this vaccine will give many a false sense of security. Maybe I am wrong but the numbers seems really low for everyone to be getting so excited over. online casino
"...those who became infected had as much virus in their blood whether they got the vaccine or a placebo — suggests that RV 144 does not produce neutralizing antibodies, as most vaccines do..."
Ummmmm....no, it doesn't: it suggests there was little CTL (=cytotoxic T-cell) induction, which is what most vaccines should do. Naughty Tony Fauci, naughty: don't speculate wildly in advance of the evidence!
NNT or NNV > 100 means IGNORE it.
Dr No and DT are right to be concerned with NNT. When it exceeds 5 or 10, it makes me skeptical. At 100, I presume it's irrelevant, entirely. But ALL of you seem to assume that HIV is a sexually transmitted disease.
It's not.
The virus called HIV might exist. It might be an HERV, i.e. endogenous in our genome already. But it's not an STD.
Seroconversions don't happen often among disparate couples (where only one gets positive HIV test results) or even in prison. AIDS is not happening in the places where STDs are happening. HIV positive tests are not happening in the places where STDs are happening. Young men and young women test HIV positive in the same proportions in the USA when they join the military.
Want more? Go see "House of Numbers" some time when you are rested and sober. Read some of the HIV=AIDS denialists books, articles, and blogs. They don't all agree on the details, but they make more sense than the orthodox believers do. Read what Clark Baker said when the Semmelweis Society asked him to investigate after they were attacked for awarding their Clean Hands award to Peter Duesberg and Celia Farber.