How effective are the mRNA covid-19 vaccines? Don’t some vaccinated people nonetheless get covid-19? Doesn’t that mean that vaccines don’t work?
Not at all. This is a logical error known as the base rate fallacy. What is the base rate fallacy?
Consider a soldier going out into battle against an army armed with bows & arrows. Doctors and experts say, “Protect yourself as best you can! Wear a suit of armor!”
This soldier went to the battlefield. Later he returns, complaining about pain in his shoulder. Here he is:
Should we conclude that a suit of armor is useless? If we look at this one person, alone, without comparing him to anyone else, it might seem that way.
But that’s illogical: there is no way to tell the effectiveness of wearing armor versus not-wearing armor by looking only at the people wearing armor.
We need to compare the rate of wounds from people in armor to the rate of wounds from people refusing to wear armor. When we do so we come to a very different conclusion:
Wow, it now is clear that when we take all the data into account, wearing armor makes one far safer.
Let’s see how this helps us understand the effectiveness of vaccines, and what it really means about breakthrough cases.
The more people in society are vaccinated, the more of the deaths will be among vaccinated people. Imagine a scenario where 95% is vaccinated, with a vaccine that is 90% effective, in a target group where IFR is 10% (80+).
In reality, as we now know, covid-19 vaccines are a lot more effective than 80%. So the infographic below even understates the advantage of being vaccinated!
In that scenario the total number of deaths among vaccinated people will be 2x higher, but their individual chance is 10x lower than that of people who are unvaccinated.
So that might sound like the vaccines are not working?
Well, that the total number is higher among the vaccinated is thus normal – it is a result of the fact that there are simply more vaccinated people around.
Similarly, if there are more red cars driving around on the roads, more red cars will crash. That doesn’t mean that the color red makes driving less safe.
You can see the actual (individual) protection provided by the vaccine on the right side, which shows the odds of dying* for vaccinated and non-vaccinated groups.
* For this fictional scenario of only 90% protection. In reality protection is a lot higher.
The vaccines do not offer 100% protection against heavy illness or death (although it is very close) .
So sometimes a (small!) part of the vaccinated people will still end up in hospital. It’s much less likely than if you are not vaccinated, but sometimes still possible.
In my infographic you see two views of the same situation: absolute numbers and odds.
– 10 unvaccinated, of which 1 died
– 190 vaccinated, of which 2 died
– 1 out of 100 vaccinated people died
– 10 out of 100 unvaccinated people died
The left box corresponds to “the numbers” that you will see in the news and on dashboards published by CDC, Worldometer, RIVM,…
But the right box is the one that counts. That one indicates that the vaccines do make a big difference.
So if you might hear about vaccinated people still ending up in hospital, keep this in mind. This is an example of the ‘Base Rate Fallacy’:
If someone is throwing percentages at you, a good reflex is to ask yourself: “exactly WHAT is this a percentage of?”
Or you could draw an example: “If I have 100 people in this group and 100 in the other, what happens to those percentages then?”
This infographic clearly shows us what group we should be comparing with what other group.
“Israel, 50% of infected are vaccinated, and base rate bias”
The statistic that’s concerning most (and that’s in the news) is a detail the Director General of the Health Ministry of Israel (Professor Chevy Levy) said during a radio interview. When asked how many of the new COVID19 cases had been vaccinated, Levy said that, “we are looking at a rate of 40 to 50%”. This must mean the Delta variant is escaping our vaccines, right? When I started digging into the numbers, though, this might not be as alarming as it seems.
This is likely an example of base rate bias in epidemiology (it’s called base rate fallacy in other fields). Professor Levy said that “half of infected people were vaccinated”. This language is important because it’s very different than “half of vaccinated people were infected”. And this misunderstanding happens all. the. time.
read more here Covid, breakthrough infections data – Your Local Epidemiologist
People are looking at the percent of vaccinated hospitalizations and getting alarmed. But by itself, this number can’t tell you much about how the vaccines are working, as it’s highly dependent on the rate of vaccination in a community. Here’s some maths to show what I mean
Making Inferences & Justifying Conclusions
Understand statistics as a process for making inferences about population parameters based on a random sample from that population.
Recognize the purposes of and differences among sample surveys, experiments, and observational studies; explain how randomization relates to each.
Evaluate reports based on data.