Join Early Retirement Today
Reply
 
Thread Tools Search this Thread Display Modes
How backtesting fails
Old 05-15-2015, 03:34 PM   #1
Thinks s/he gets paid by the post
photoguy's Avatar
 
Join Date: Jun 2010
Posts: 2,301
How backtesting fails

I thought this was a very good article (although somewhat technical) about how backtesting can fail or mislead:

9 Mistakes Quants Make that Cause Backtests to Lie by Tucker Balch, Ph.D. | Quantopian Blog

I never did any backtesting with financial/stock data (beyond running FIRECALC etc), but I saw a lot of these errors in other areas during my former work life. These were even made by people with advanced degrees in quantitative fields.

Just something to think about for those trying to interpret backtesting studies or running their own tests...
__________________

__________________
photoguy is offline   Reply With Quote
Join the #1 Early Retirement and Financial Independence Forum Today - It's Totally Free!

Are you planning to be financially independent as early as possible so you can live life on your own terms? Discuss successful investing strategies, asset allocation models, tax strategies and other related topics in our online forum community. Our members range from young folks just starting their journey to financial independence, military retirees and even multimillionaires. No matter where you fit in you'll find that Early-Retirement.org is a great community to join. Best of all it's totally FREE!

You are currently viewing our boards as a guest so you have limited access to our community. Please take the time to register and you will gain a lot of great new features including; the ability to participate in discussions, network with our members, see fewer ads, upload photographs, create a retirement blog, send private messages and so much, much more!

Old 05-15-2015, 04:07 PM   #2
Thinks s/he gets paid by the post
2017ish's Avatar
 
Join Date: Apr 2012
Posts: 1,845
Thanks. Interesting, and I would not have been likely to run across it.
__________________

__________________
OMY * 3 2ish Done 7.28.17
2017ish is offline   Reply With Quote
Old 05-15-2015, 04:14 PM   #3
Thinks s/he gets paid by the post
Onward's Avatar
 
Join Date: Jul 2009
Posts: 1,664
Good article and interesting website. Thanks.
__________________
And if I claim to be a wise man, it surely means that I don't know.
Onward is online now   Reply With Quote
Old 05-15-2015, 06:49 PM   #4
Thinks s/he gets paid by the post
 
Join Date: Oct 2006
Posts: 3,815
Interesting article, Thanks.

I was surprised by this:

" If, instead we use the members of the S&P 500 starting in 2008, we find that more than 10% of the listed companies failed."

Do you know if "failed" means "went bankrupt"? or "dropped out of the S&P 500"?
__________________
Independent is offline   Reply With Quote
Old 05-16-2015, 12:23 PM   #5
Thinks s/he gets paid by the post
photoguy's Avatar
 
Join Date: Jun 2010
Posts: 2,301
When I first read the article I thought the 10% figure was a hypothetical. But I think after some googling that the writer is referring to the number of companies removed for distress reasons (see attached figure from S&P 500 Deletions Due To Distress - Business Insider ). The removals from distress seem to match his 10% figure.

From the article
Quote:
the S&P 500 deletions that were a consequence of stocks that failed outright, were removed due to substantial declines in their market value, or were acquired after suffering such a decline
Attached Images
File Type: jpg distress.jpg (297.8 KB, 15 views)
__________________
photoguy is offline   Reply With Quote
Old 05-16-2015, 12:47 PM   #6
Recycles dryer sheets
 
Join Date: Jun 2014
Posts: 433
That's pretty neat. I just looks at the richest lists an don't see many (any) quant traders on those lists so I figure... Can't work that well

Sent from my HTC One_M8 using Early Retirement Forum mobile app
__________________
petershk is offline   Reply With Quote
Old 05-16-2015, 06:46 PM   #7
Thinks s/he gets paid by the post
 
Join Date: Oct 2006
Posts: 3,815
Quote:
Originally Posted by photoguy View Post
When I first read the article I thought the 10% figure was a hypothetical. But I think after some googling that the writer is referring to the number of companies removed for distress reasons (see attached figure from S&P 500 Deletions Due To Distress - Business Insider ). The removals from distress seem to match his 10% figure.

From the article
Thanks. That's a higher number than I would have guessed. Nice chart.
__________________

__________________
Independent is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Middle School Teacher Starts Business, Fails and Hurts a Lot of People haha FIRE and Money 15 09-29-2008 03:18 PM
Who protects annuities if the insurer fails? Maurice FIRE and Money 67 09-17-2008 09:50 AM
Allocation backtesting trixs Other topics 3 03-10-2008 09:12 PM
GMail notifier mailto function fails in Firefox TromboneAl Other topics 2 01-28-2008 04:56 PM
Homes wash away after Missouri dam fails REWahoo Other topics 4 12-14-2005 02:03 PM

 

 
All times are GMT -6. The time now is 11:39 PM.
 
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.