Lottery winners and stock returns: the problem of survivor bias

A lottery winner addresses his audience and implores them to never stop buying lottery tickets, no matter what anyone tells them.

“I failed again and again, but I never gave up,” he tells them. “I took extra jobs and poured the money into tickets. And here I am, proof that if you put in the time, it pays off!”

The above sketch, which appears in the XKCD webcomic, is a humorous example of an illusion that pervades the investment world – survivorship bias, the tendency to focus on successes (in this case, the lottery winner) while ignoring past failures (the millions of people who buy tickets and win nothing).

Managed funds

Index fund giant Vanguard describes survivorship bias as "one of the most common – and momentous – flaws in data analysis". A Vanguard study from 2013 found that 62 per cent of large-cap value funds had beaten their benchmark index over the previous five years.

Are record corporate profit margins sustainable?

---

Value investing made Buffett rich, but will it work for you?

---

Musk’s behaviour should concern Tesla investors

---

That sounds impressive, but that statistic only covers funds that survived the full five-year period. It’s common for poorly-performing funds to be shut down. By excluding the non-survivors, it’s reasonable to assume that the results seem better than they really are – an assumption borne out by Vanguard’s analysis, which confirmed the overwhelming majority of closed funds were underperformers.

After including these closed funds, Vanguard found the number of outperformers over a five-year period fell from 62 to 46 per cent.

Over a 10-year period, the percentage of outperformers fell from 48 to 32 per cent. The figures were even more stark over a 15-year period, with the number of outperformers falling from 50 to just 27 per cent. The figures are similar for growth funds, with the percentage of outperformers over a 15-year period falling from 34 to 18 per cent once the closed funds are included.

Vanguard’s analysis is echoed by the University of Chicago’s Center for Research on Securities Prices, which examined the performance of funds over a 50-year period. It found the average fund investor would have seen a $100 investment grow to about $2,500 over the lifetime of their investment.

However, those figures pertain to surviving funds only – an incomplete picture, given that roughly 13,000 of the 39,000 funds analysed died over the period. Once these funds are included, the $100 investment grows to $2,100. In other words, not adjusting for survivorship bias inflates the returns enjoyed by investors by 19 per cent.

Incubated funds

English mentalist and television personality Derren Brown once correctly predicted he could toss a coin 10 times in a row and have it come up heads every time. However, Brown admitted this wasn't skill or magic on his part. He simply filmed himself tossing coins thousands of times until he finally (it took nine hours) got the streak he wanted.

Fund management companies perform a similar trick known as fund incubation. A number of different funds are set up and run privately, using a relatively small amount of money raised internally. If one fund delivers good returns, it can then be sold to the public, with the key marketing point being the stellar performance produced during the incubation stage. No one ever hears of the poor performers, of course, which never see the light of day and are quietly wound down.

Fund managers might argue that this is harsh and that incubation can be used to identify skilled managers or sound strategies. However, research suggests otherwise. One 2015 study noted that incubated funds tend to be launched to the public “at the first sight of outperformance”.

Unfortunately for the investors who buy into the fund on the basis of its track record, performance typically “reverses in the post-incubation period, which supports the view that incubation is not used to identify skilled managers, but to establish a track record of artificial outperformance”.

Global indices

US stocks have averaged annualised returns of about 10 per cent over the last century and have never experienced a 20-year period where they failed to beat inflation. However, the notion that stocks are inevitably a good long-term bet ignores the fact that equities have not always survived troubled periods.

The Russian stock market was one of the biggest in the world at the start of the 20th century. It accounted for roughly 6 per cent of global stock market capitalisation, having easily outperformed the US market in the 50 years after the end of the American civil war. By 1917, however, Russian investors had been wiped out, with the market not reopening until the 1990s. Investors in China, too, lost all of their capital after the communists took control in 1949.

Triumph of the Optimists: 101 Years of Global Investment Returns, the classic book written by London Business School researchers Elroy Dimson, Paul Marsh and Mike Staunton, notes that financial researchers have tended to focus on countries where data could be easily obtained. An index may be initiated "after unrest, or wars, and their aftermath have been resolved", providing investors with a "misleadingly favourable" impression of long-term stock returns.

Business books

Survivor bias contaminates the findings of some of the bestselling business books of recent decades. In his 2001 bookGood to Great, management guru Jim Collins focused on 11 stocks that had trounced the market and then searched for the shared traits he deemed to be responsible for their success.

However, critics pointed out many not-so-successful companies also shared some of these traits, which included a "culture of discipline" and "level five leadership". In his book Standard Deviations, economist Gary Smith describes Collins's work as a "backward-looking study undermined by survivor bias" and notes that a portfolio consisting of the 11 stocks went on to deliver "distinctly mediocre" returns over the following decade. One of the chosen stocks, mortgage lender Fannie Mae, had to be taken over by the US government in 2008. Another, Circuit City, went bankrupt in 2009.

Smith complains that the same wrong-headed approach was used by Tom Peters and Robert Waterman in their 1982 bookIn Search of Excellence. Hugely influential and an international bestseller, it focused on 43 apparently great companies that shared eight common traits. Again, there was no evidence these traits – for example, a bias for active decision-making, being close to the customer, and entrepreneurship – were responsible for their previous success. A majority of the firms, which included soon-to-be-forgotten companies such as Atari, Data General, DEC, NCR and Wang Labs, underperformed in subsequent years.

“This problem plagues the entire genre of books on formulas/secrets/recipes for a successful business, a lasting marriage, living to be 100, and so on and so forth, that are based on backward-looking studies of successful businesses, marriages and lives,” says Smith. “There is inherent survivor bias.”