The base rate fallacy

As testing for Covid-19 immunity becomes widely available, it's important to understand the "Base Rate Fallacy" that leads people who test positive to reach false conclusions about the true probability that they are actually immune. 

Let's assume that 10% of the population is immune. Epidemiologists refer to this as the population's "base rate" of immunity. If the base rate is 10%, there's a 10% chance that a randomly-selected individual is immune. 

Now test that individual for immunity with a test known from clinical trials to result in 5% false positives. this means that 5% of the time, individuals testing immune actually are not. Let's also assume that the test doesn't result in any false negatives, meaning that everyone who is immune will test immune. 

When we use this test on our randomly selected individual and she tests positive (immune), what is now the probability that she really is immune?

The answer is that a positive test changes the probability of her being immune from 10% to 69%, but not to 95%, as intuition might suggest from a 95% accurate test.  

The reason why is explained by the "base rate" of 10%.

Our randomly-selected individual doesn't know whether she is actually immune. Before taking the test, all she knows is that there's a 10% chance she is immune.

Now since 90% of the population really isn't immune, and the test is 95% accurate, then 5% of this 90%, or 4.5% of the population, will test immune even though they are not.

For the 10% of the population that really is immune, the test is 100% accurate so 100% of the 10% will test immune, representing 10% of the population.

So how many people will test immune in all?  4.5% of the population will falsely test positive, while 10% will truly test positive. So overall there will be 14.5% positive tests.

Of these 14.5%, 4.5% are false.  That means that when the test is applied to a population having a "base rate" of immunity of 10%, the false-positive rate will 4.5 divided by 14.5, or 31%. So, the probability that a randomly-selected indivdual who tests positive really is positive is 69%.

This chart shows the relationship between the base rate (X-axis) and the probability that someone testing immune really is immune (Y-axis).  The blue line shows the relationship for a test having a 5% false-positive rate in clinical trials, and the red line shows the relationship for a test having a 10% false-positive rate.

Finally, note that when the base rate is 50% (at the far right of the chart), the probability of a false positive for a randomly-selected invidiual is the same as the test's false-positive rate in clinical trials.