Expected value and base rates

Weigh outcomes by frequency, and stop reading a test result as a probability.

A test is 90% accurate. You test positive for a disease that one person in a hundred has. Most people, including most doctors when the question is put this way, answer that you probably have it. The right answer is that there is about a one in twelve chance, and the reason has nothing to do with the test being bad.

Two things, one idea

Expected value is the average result if a choice were made many times: each outcome multiplied by how often it happens, added up. It is the thing people mean when they say a bet is worth taking.

Base rates are how often something is true before you looked. They are the expected value of a belief, and they are the number people abandon first when a vivid piece of evidence arrives.

The two are the same instruction. Weigh what you are looking at by how often it happens, and do not let one striking case stand in for a frequency.

The test, in whole people

A thousand people below. Ten have the disease. The test catches nine of them and wrongly flags 99 of the healthy 990.

Has it, test says so 9
Has it, test missed it 1
Does not have it, test says they do 99
Does not have it, test agrees 891

108 people test positive. 9 of them have it.

8.3%chance you have it, given a positive result

The test still did something. It moved your odds up by about 8 times. Moving the odds and answering the question are different jobs.

How common it is1.0% of peopleCatches it when it is there90%Clears you when it is not90%

Now make the disease commoner and watch the answer climb, then make the test better and watch it climb much less. How rare the thing is does more work than how good the test is, which is the opposite of where attention usually goes.

Worked example

Where the intuition goes wrong

The mistake is answering a question that was not asked. "The test is right 90% of the time" is a fact about the test: given that you have it, it says so 90% of the time. The question you care about runs the other way: given that it said so, do you have it?

Count the two groups of positives separately and it comes apart cleanly.

9 true positives. Ten people have it, the test catches nine.
99 false positives. Nine hundred and ninety do not have it, and the test is wrong about 10% of them.
108 positives in total, 9 of which are real. That is 8.3%.

The false alarms outnumber the real cases eleven to one because there are ninety-nine times more healthy people to be wrong about. Nothing about the test needs to be poor for this to happen. It happens because the disease is rare.

And the test was not useless. You went from a 1% chance to an 8.3% chance, which is more than eight times more likely. That is what a test does: it moves the odds. Treating the move as an answer is the error.

The arithmetic is checked by tests, including one that runs the counts and the unrounded probability down separate paths and requires them to agree. The natural frequency presentation is from Gerd Gigerenzer, who showed that doctors who get this wrong in percentages mostly get it right when the same problem is stated in whole people.

ConventionStatisticians call the starting frequency the prior and the updated one the posterior, and the rule for moving between them is Bayes' theorem. You do not need the formula. You need the habit of asking what the number was before the evidence arrived, because the formula is doing nothing more than refusing to throw that number away.

Expected value, and where it stops applying

Expected value is the right tool when the same decision repeats and you can survive the variation. An insurance company sells a hundred thousand policies and lands very close to the average. That is their whole business model.

It is the wrong tool when the decision happens once and a bad outcome removes you from the game. A bet with a positive expected value and a 10% chance of bankruptcy is a bad bet if you only get to make it once, because there is no long run in which the average shows up for you.

This is why the same arithmetic recommends buying and buying insurance, which look like opposite decisions. Index funds are the repeated bet where the average arrives. Insurance is the single event where it does not, and you pay a little to move the tail onto somebody with a hundred thousand policies.

What breaks

Confusing a vivid case with a frequency. A story about somebody who won feels like evidence and is a sample of one, drawn from the winners, by somebody who selected it for being tellable. Ask how many people did the same thing and are not being written about.

Applying a base rate from the wrong group.The right base rate is the one for the narrowest group you genuinely belong to. "Most restaurants fail" is a worse starting point than "most restaurants opened by people with ten years in the business, in this city, fail", if you can find the second number.

Treating an unknown probability as 50-50. Two outcomes does not mean two equal outcomes. Not knowing the odds is a reason to go and estimate them, and calling them even is a specific and usually wrong claim rather than a neutral one.

What to do with this

Ask two questions before you act on any piece of evidence. How often is this true anyway? And how much does what I just learned actually move that?

Most evidence moves it less than it feels like it should, and knowing roughly how much is worth more than knowing the formula. A test, a signal, an interview, a quarter of good results: each of them shifts the odds by some amount, and the shift is almost never all the way to certain.

Test yourself

01A coin flip pays $120 if it comes up heads and costs you $100 if it does not. Should you take it once? Should you take it a thousand times?

The expected value is $10 a flip either way, so on that measure it is a good bet both times. But once and a thousand times are genuinely different questions, and the difference is not about the average.

A single flip has a coin-flip chance of costing you $100, which matters a great deal if $100 is your rent. A thousand flips almost certainly nets you around $10,000, because the variation washes out. Expected value tells you which side of zero you are on. Whether you can afford the spread around it is a separate question, and losing everything on the way to a good average is not a rounding error.

02A startup founder says nine in ten startups fail, but theirs is different because the team is excellent. What is wrong with the reasoning, and what would fix it?

Nothing is wrong with adjusting away from the base rate. The mistake is discarding it. Nine in ten is where you start, and evidence moves you off it by however much that evidence is actually worth.

The fix is to ask what the failure rate is among startups with excellent teams, which is the base rate for the group they are actually in. It is lower than nine in ten and it is not close to zero, because most failed startups also had teams that looked excellent at the time.

03Why does a test that is 99% accurate still give mostly wrong positives for a rare condition?

Because the false alarms are drawn from a much larger pool. With a condition affecting 1 in 10,000, a million people contain 100 real cases and about 10,000 false alarms even at 99% specificity. The test is doing what it says. There are simply a hundred times more healthy people to be wrong about, and being wrong 1% of the time about a lot of people beats being right 99% of the time about very few.

A tutor that knows this lesson. It asks before it explains, and it will not tell you what to do with your own money.

Educational material, not investment or policy advice. Figures are cited where they come from a filing or a statistical series, and labelled as illustrative where they do not.