How backtesting fails

[photoguy]

I thought this was a very good article (although somewhat technical) about how backtesting can fail or mislead:

9 Mistakes Quants Make that Cause Backtests to Lie by Tucker Balch, Ph.D. | Quantopian Blog

I never did any backtesting with financial/stock data (beyond running FIRECALC etc), but I saw a lot of these errors in other areas during my former work life. These were even made by people with advanced degrees in quantitative fields.

Just something to think about for those trying to interpret backtesting studies or running their own tests...

 

[2017ish]

Thanks. Interesting, and I would not have been likely to run across it.

 

[Onward]

Good article and interesting website. Thanks.

 

[Independent]

Interesting article, Thanks.

I was surprised by this:

" If, instead we use the members of the S&P 500 starting in 2008, we find that more than 10% of the listed companies failed."

Do you know if "failed" means "went bankrupt"? or "dropped out of the S&P 500"?

 

[photoguy]

When I first read the article I thought the 10% figure was a hypothetical. But I think after some googling that the writer is referring to the number of companies removed for distress reasons (see attached figure from S&P 500 Deletions Due To Distress - Business Insider ). The removals from distress seem to match his 10% figure.

From the article

the S&P 500 deletions that were a consequence of stocks that failed outright, were removed due to substantial declines in their market value, or were acquired after suffering such a decline

 

[petershk]

That's pretty neat. I just looks at the richest lists an don't see many (any) quant traders on those lists so I figure... Can't work that well

Sent from my HTC One_M8 using Early Retirement Forum mobile app

 

[Independent]

When I first read the article I thought the 10% figure was a hypothetical. But I think after some googling that the writer is referring to the number of companies removed for distress reasons (see attached figure from S&P 500 Deletions Due To Distress - Business Insider ). The removals from distress seem to match his 10% figure.

From the article

Thanks. That's a higher number than I would have guessed. Nice chart.