J&J vaccine and Guillain-Barre syndrome: a look at the probabilities
--
Or the relationship between vaccine adverse effects and lottery winners.
Disclaimer : I’m not in the medical field and this article is not about the risk or benefits of vaccines. I just like to understand the numbers that get thrown around in the media.
A few days ago, the FDA announced new findings about a possible side effect of the J&J Covid-19 vaccine. This news was quickly picked up by several media. This gist of it consists of :
There have been 100 preliminary reports of GBS (Guillain-Barre syndrome) following vaccination with the Janssen vaccine after approximately 12.5 million doses administered.
When I first read it, it went through 3 phases in a matter of seconds :
- “100 cases, that’s a lot !”
- “Wait… 100 cases over 12.5 million, that’s almost nothing, right?”
- “ Oh boy, I have no idea what that means”
Occurrence rate
So I decided to make some sense of it. The first step was to ask :
What number of GBS cases should we have expected ?
For that, I need to know the usual occurrence rate for GBS. It varies from source to source but I will settle on 1.5 cases per 100.000 people per year (or a 0,0015% chance).
The J&J vaccine being a “one-shot” vaccine, means that the 12.5 million doses translate to 12.5 million people vaccinated. So how many CBS cases could I have expected?
Strange. That’s higher than the 100 cases from the news. What’s the fuss? Well, the J&J vaccine rollout started only in March 2021, not a year ago. And the vaccinations did not all occurred early March, they were distributed across several months. So I need to know how many days ago, on average, a patient has received a dose.
OurWorldInData has data detailing the cumulative number of doses administered for each manufacturer in the US. The 12.5 million mark has been reached on July 2nd, so I will discard anything after. With a little work, I can compute the daily number of administered doses.
For example, 315.481 people received their doses on March 13th. And there are 111 days between March 13th and July 2nd. Using the same pattern, I can compute the weighted average of days between injection and our cut-off date: 72 days.
My expected occurrences become:
So 37 versus 100 (so an x2,7 factor). Now I see the problem.
To put things in perspective, this is still a very low probability. That basically means that taking the J&J vaccine would give a 0,00162% additional chance of developing GBS.
Discarded factors: obviously this is just an approximation. To have something more accurate, I need to know how long it takes for GBS to be diagnosed. I also assumed that the people vaccinated are representative of the general population (they are not, as the early vaccination effort targeted mostly older and vulnerable people), and maybe the GBS rate is affected by those criteria. And, most importantly, there already seems to be an increased likelihood of GBS in COVID-19 patients. As the vaccine does not totally prevent infection, it could be that the increase of GBS seen in J&J vaccinated people is in fact (partially or totally) due to COVID-19 infection, not the vaccine.
How lucky are you?
Having 100 cases instead of 37 seems significant. But how significant?
Let’s say I pick 100.000 people at random in a given year. Surely I’m not expecting to have exactly 1,5 cases of GBS (how would that even work?). I can expect 1 or 2. But what about 3? Or 5? Or 0? This still seems possible.
So what about our 37 expected cases? Could they have been 100 just by luck? In other words:
What is the probability of having 100 cases of GBS when we are expecting 37?
Without more information than the base occurrence rate, it’s hard to create a proper model. So I’ll have to resort to an approximation. A Poisson distribution could be a good fit, as it is intended for a distribution of individual events where only the mean is known.
Poisson gives us a simple formula to find the probability of a number k of events occurring when the expected mean is λ.
So the probability of having 100 cases when I was expecting only 37 is :
This is very very low. In fact, you would have more chance of winning the lottery.
A frequent rare event
Speaking of lottery, let’s observe something: winning is extremely unlikely. 1 chance in 292,201,338 for the Powerball. Yet, it has been won 7 times in 2020. Individually, each winner “shouldn’t” have won, so why are we not surprised when we hear that someone wins?
Because a lot of people play.
This begs the question: how many players are there at the “rare disease” lottery?
To know how many rare diseases there are, we first need to define what is a “rare” disease. This definition change from country to country but the European Union has a fairly simple one: any disease that has a rate of occurrence of less than 1 per 5.000 people per year. And that means there are about 8.000 “rare” diseases.
So, is there a real problem with GBS, or is it simply an unfortunate winner. In other words:
If you pick 12,5 million people at random for 72 days, what are the odds that you find an abnormal (more than double) number of occurrences in at least one of the 8.000 rare diseases?
I’ll spare you the details, I ended up with this formula :
The problem is to choose the rate to use.
Per the EU’s definition, it starts at 1 per 5.000. But it can go way lower than that. GBS is at 1~2 per 100.000 (or 1 per 66.667) and some diseases can be even rarer. I did not find enough data to compute the average (or geometric mean) of the occurrence rates for rare diseases, and I don’t want to risk myself guessing one when there is so much discrepancy.
So I’ll approach the situation from the other side. I’ll try to find the average occurrence rate that would make having at least one “doubling” disease be plausible (i.e. more than a 5% probability, to match the most common P-Value used in medical studies)
The simplest way to read this graph is to look at the 5% probability mark: to have at least a 5% chance of observing a “natural” doubling of rate in one of the rare diseases for a population of 12.5 million over 72 days, the average rate would need to be at 1 case per 105.000 per year.
To have a very good probability (> 50%) would require an average rate of about 1 case per 145.000 per year.
That’s quite far from the GBS rate, and very far from the entry rate for rare diseases. So I think we can conclude that, in our situation, a natural doubling of cases would be unlikely. And that the most logical conclusion is that a correlation indeed exists between GBS and the J&J vaccine.
Sample size and duration
Let’s end with a few other questions.
Could we have seen this during the clinical trial?
The obvious answer is no. GBS has a base rate of 1,5 per 100.000. The trial was composed of two arms: 21,895 vaccinated people and 21,888 non-vaccinated people. They were observed during 58 days (median).
So, the expected number of cases of GBS in each group would have been… 0,05. Even with our increased rate, there would have been most likely no cases.
What kind of occurence rate could the trial have detected?
The problem is GBS is exceptionally rare. Its rate would need to be multiplied by more than 13 to become “common” (i.e. a rate of more than 1 per 5.000 per year).
The trial population size and duration allow us to reliably observe cases of diseases when their rate is at least 1/1.000. So we could conclude that a study can only detect adverse effects that have a rate of about 1/1.000 per year and cannot detect rare diseases at all.
That would be the case if the vaccine had mostly long-term adverse effects (i.e. it would increase the likelihood of a given disease forever, in a constant fashion). But most adverse effects of the vaccines are short-term: instead of a permanent increase of the rate of the disease, we would see a sudden burst a few days or weeks after injection.
For this kind of adverse effect, the “short” duration of the trial is less problematic and the trial is large enough to reliably detect “rare” (1/5.000) events. Obviously, for very rare events (like GBS) 21,895 people are simply not enough.
What about the Pfizer trial?
The Pfizer study was done on a similarly sized population (21,720 with BNT162b2 and 21,728 with placebo). The duration was longer with most of the patients being monitored for two months after the second shot (and some for more than 3 months). With 21 days between the two shots, it means adverse effects could have been observed for about 81 days.
The longer duration improves the detection of long-term diseases, but not by much. For short-term effects, the precision is similar to the J&J trial.
Conclusion
As I explained in the Discarded factors early on, this is a very simple analysis. I’ve left out a lot of factors that could paint a whole other story between PBS and this vaccine.
At least, I understand now why the alarm has been raised, and why this possible adverse effect could not have been seen during the trial (or the first months of the roll-out).
Hopefully, we will have more information soon and we will be able to know if there is causation (and not simply correlation) between this vaccine and the Guillain-Barre syndrome.
If I somehow got my maths wrong, feel free to comment :)
