The Unknown Unknowns and Survivor Bias

Survivorship bias helps us understand why success stories are not what actually help us succeed

by charles smith

Former Secretary of Defense Donald Rumsfeld is famous for uttering a koan-like description of the epistomological ambiguity of human experience:

There are known knowns; there are things we know that we know.

There are known unknowns; that is to say, there are things that we now know we don't know.

But there are also unknown unknowns – there are things we do not know we don’t know.

(Interestingly, it appears Rumsfeld did not pen the koan himself; correspondent J.S.S. noted that the original source may be Landmark Education of Seattle, Washington.)

I recently read two fascinating accounts of why we have such a difficult time knowing what we don't know: it's called survivor bias, and what that means is we only get information from the survivors, not those who perished and vanished from the records.

A mid-list author recently explained how listening to the handful of authors who make it big financially is completely misleading: Survivorship bias: why 90% of the advice about writing is BS right now.

Here's how this works: big-bucks Author Z says, here are the 10 steps you need to take to become as successful as me. The list always mentions perseverance, being nice to your readers, writing 1,000 words a day and so on.

The 99.9% of writers/authors who make less than $10,000 a year from their writing (and the 99% who make less than $1,000) take this list as a script or program that if followed, will yield great success. But the 99% follow the script and do the 10 things and discover they are still unknown and not making any money.

The point is that nobody asks the 99% who fail to make it big what they did, or try to analyze what they did that prevented their success.

The numbers of people who become successful in these sorts of high-competition careers is vanishingly small--hedge fund managers are an example, along with musical acts, artists and writers. A handful at the top make most of the money, a relative few make some money (let's say a middle-class income) and then 99% make near-zero.

Longtime correspondent B.C. recently sent me an analysis of how many mutual fund managers outperformed a "dumb" low-cost index fund. B.C. found that 66 managers out of 26,507 outperformed the SPX over 5 years. That mirrors the distribution of outsized success among hedge fund managers, a few of whom net $500 million each annually while the rest underperform a plain old index fund.

The conclusion is that luck is the ultimate factor in these signal-noise levels of success, and being in the right place at the right time with the right product/idea can't be replicated by following a script or list of tips.

This essay is longer and well worth a read: Survivorship Bias The Misconception: You should study the successful if you wish to become successful.

The basic idea here is that studying the aircraft that come back riddled with holes from bombing missions leads us astray when we try to analyze the damage: what would really help us is studying the planes that were shot down, but they are unavailable for study.

The same principle applies to restaurants: the few that become roaring successes are endlessly studied, but the causes of success and failure are actually buried in the stories of all those that failed and close their doors. But nobody collects that data, for a number of reasons, including the study of failure isn't sexy and won't sell magazines.

We know that we learn from mistakes and failures, yet the study of failure is never recorded or saved unless the company or individual "came back from the dead," for example, Apple after Steve Jobs returned to lead the company from the abyss in 1996.

Survivorship bias is a profound insight into how we confuse the known unknowns and the unknown unknowns.