Join Early Retirement Today
Reply
 
Thread Tools Search this Thread Display Modes
Monte Carlo Simulations-now recognized as understating risk
Old 05-02-2009, 11:46 PM   #1
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
 
Join Date: May 2004
Posts: 11,615
Monte Carlo Simulations-now recognized as understating risk

The WSJ had a piece this weekend on Monte Carlo simulations, and the fact that now (surprise!!) many people believe the simulations understate the risks of real world investing.

Some portions:
Quote:
If one had asked a financial adviser 18 months ago for retirement-planning guidance, there is a good chance he would have run a "Monte Carlo" simulation. . .
But there is little chance your Monte Carlo simulation would have highlighted a scenario like the market slide just seen. Though these tools typically run a portfolio through hundreds or thousands of potential market scenarios, they often assign minuscule odds to extreme market events. Yet these extreme events seem to be happening more often.
. . .
Monte Carlo simulation has wide appeal, and is used in online tools offered by firms like Fidelity Investments and by independent retirement planners.
. . .

Many providers of the tools argue that it is a significant improvement over the traditional retirement-planning approach, which typically involves assuming some set market return, say 8% for U.S. stocks, year after year, an assumption considered unrealistic by academics and financial pros. The questions about Monte Carlo tools reflect broader concerns about mathematical models for gauging portfolio risks. . . .
These models were supposed to help quantify and manage the risks of mortgage-backed securities, credit-default swaps and other complex instruments. But given the events of the past couple of years, it appears that the models often gave big institutions, as well as small investors, a false sense of security.
. . . .
Now, some investors have decided that if risk can't be accurately measured, they will just have to play it safe. . . .
Some financial advisers are equally skeptical. "I take whatever probability of failure that comes out of your Monte Carlo simulation and add 20 percentage points," said William J. Bernstein, author of "The Four Pillars of Investing."
Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it. Since no standard approach exists, one user might plug in a range of assumptions on interest rates, inflation or volatility that is different from another user.
Also controversial is that many Monte Carlo simulations assume that market returns fall along a bell-curve-shaped distribution. . . .
"In a bell-shaped curve the probability of getting one of these extreme outcomes we're seeing is basically zero," said Paul Kaplan, vice president of quantitative research at Morningstar Inc.
. . . .
While a bell-curve model indicates there is almost no chance of a greater than 13% monthly decline in the Standard & Poor's 500-stock index, such declines have happened at least 10 times since 1926, according to a report by Mr. Kaplan.
Some Monte Carlo models, like the one used by Financial Engines, assign higher odds to extreme market events than the bell-curve distributions. . . .

Some firms are considering revising Monte Carlo models to reflect a world where big market swings happen more often. Morningstar last year tweaked its asset-allocation software offered to institutional investors, allowing users to choose a bell-curve-shaped distribution or a "fat-tailed" distribution, which assigns higher probabilities to extreme market events. The company is exploring using this model in more products, Mr. Kaplan said.
Laurence Kotlikoff, a Boston University economics professor who developed the ESPlanner financial-planning software, and Richard Fullmer, senior portfolio strategist at Russell Investments, said they also are considering offering clients Monte Carlo scenarios that incorporate fatter-tailed distributions. . . .

Some industry participants also are trying to set standards that could help Monte Carlo tools more accurately capture extreme market events. The Retirement Income Industry Association in 2007 issued a set of principles noting that the calculators should run a large number of scenarios.
The ideal models run tens of thousands or hundreds of thousands of scenarios, which help gauge extreme events at the tail end of the distribution, observers said. Yet some tools run only 1,000 scenarios or just several hundred.
It's good that MC simulations are getting more attention. Still, this article didn't do a good job of explaining some of the common weaknesses (esp capturing complex interdependencies among asset classes and adjacent-year returns, and instead treating them as independent. This significantly understates risk).

As Bernstein has pointed out in the past, there's systemic risk (including the fairly regular collapse of governments and economic systems) that overwhelms some of the miniscule failure rates predicted by these MC simulators.

Also, too bad they didn't touch on FIRECalc.
__________________

__________________
"Freedom begins when you tell Mrs. Grundy to go fly a kite." - R. Heinlein
samclem is online now   Reply With Quote
Join the #1 Early Retirement and Financial Independence Forum Today - It's Totally Free!

Are you planning to be financially independent as early as possible so you can live life on your own terms? Discuss successful investing strategies, asset allocation models, tax strategies and other related topics in our online forum community. Our members range from young folks just starting their journey to financial independence, military retirees and even multimillionaires. No matter where you fit in you'll find that Early-Retirement.org is a great community to join. Best of all it's totally FREE!

You are currently viewing our boards as a guest so you have limited access to our community. Please take the time to register and you will gain a lot of great new features including; the ability to participate in discussions, network with our members, see fewer ads, upload photographs, create a retirement blog, send private messages and so much, much more!

Old 05-03-2009, 12:06 AM   #2
Moderator Emeritus
Nords's Avatar
 
Join Date: Dec 2002
Location: Oahu
Posts: 26,617
But... but... but... this would mean that returns aren't log-normal distributions!

Apparently quite a bit has been lost in translation between the research papers and the mutual-fund company websites.
__________________

__________________
*
*

The book written on E-R.org, "The Military Guide to Financial Independence and Retirement", on sale now! For more info see "About Me" in my profile.
I don't spend much time here anymore, so please send me a PM. Thanks.
Nords is offline   Reply With Quote
Old 05-03-2009, 06:37 AM   #3
Full time employment: Posting here.
 
Join Date: Oct 2007
Location: New York
Posts: 898
This is Nicholas Nassim Taleb's 'black swan' point exactly, applied to MC analyses.

As for how important it is, it depends on how the MC analysis is being used. In many banks, MC analyses are used to value many derivatives with path-dependent valuations. In this case using a gaussian distribution is probably folly.

But in many fire-calc type analyses you use the gaussian distribution but then set such a high bar on your planning (say 95% survival rate) that you're really giving much more weight to bad events than is implied through the use of the gaussian model.
__________________
Money's just something you need in case you don't die tomorrow.
Maurice is offline   Reply With Quote
Old 05-03-2009, 07:30 AM   #4
Dryer sheet wannabe
 
Join Date: Nov 2008
Posts: 23
I always wondered about the logic of using something like the 95% success rate. That means that in the last 70-100 years (depending on your data set), your worst case scenario did happen. Considering the down side-- having no personal resources in your helpless advanced old age-- that doesn't seem very prudent. Picking 100% at least attempts to factor in *all* catastrophic economic events within living memory. I'm a Taleb fan, obviously.
__________________
rjsciurus is offline   Reply With Quote
Old 05-03-2009, 07:34 AM   #5
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
brewer12345's Avatar
 
Join Date: Mar 2003
Posts: 16,391
Quote:
Originally Posted by rjsciurus View Post
I always wondered about the logic of using something like the 95% success rate. That means that in the last 70-100 years (depending on your data set), your worst case scenario did happen. Considering the down side-- having no personal resources in your helpless advanced old age-- that doesn't seem very prudent. Picking 100% at least attempts to factor in *all* catastrophic economic events within living memory. I'm a Taleb fan, obviously.
100% doesn't mean you are safe, either. The future may (quite likely, IMO) contain something worse than the historical record.

You pays your money and you takes your chances, same as always, Taleb's obnoxious commercializing of this central concept notwithstanding.
__________________
"There are three kinds of men. The one that learns by reading. The few who learn by observation. The rest have to pee on the electric fence for themselves."



- Will Rogers
brewer12345 is offline   Reply With Quote
Old 05-03-2009, 07:34 AM   #6
Full time employment: Posting here.
 
Join Date: Oct 2007
Location: New York
Posts: 898
Yeah, I like the analyses that show actuals from the late 1800s on, its a nice compliment to MC analyses.

On the other hand, most of these models don't take into account the fact that our reactions to changing markets are dynamic. Most of us don't continue taking out the same amount in bad years - we cut back and take less if we can.
__________________
Money's just something you need in case you don't die tomorrow.
Maurice is offline   Reply With Quote
Old 05-03-2009, 07:46 AM   #7
Recycles dryer sheets
 
Join Date: Mar 2005
Posts: 58
It also says that by using tens or hundreds of thousands of iterations, the accuracy is improved.
FIRECalc doesn't use that many interations and does not incorporate "fatter tailed distributions".
__________________
mark is offline   Reply With Quote
Old 05-03-2009, 07:49 AM   #8
Administrator
W2R's Avatar
 
Join Date: Jan 2007
Location: New Orleans
Posts: 38,877
Quote:
Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it. Since no standard approach exists, one user might plug in a range of assumptions on interest rates, inflation or volatility that is different from another user.
Also controversial is that many Monte Carlo simulations assume that market returns fall along a bell-curve-shaped distribution. . . .
Modeling is fun, and people without sufficient modeling background frequently use (mis-use) models as described. This is nothing new. Poor problem selection is an egregious mis-use of models. Poor interpretation of results that were obtained based on poor problem selection and inappropriate assumptions is simply ignorant, IMO.

Financial planning problems involve a huge numbers of unknowns and assumptions and also unknown probabilities ("past performance is no guarantee of future results" points out just one of many possible bad assumptions). That doesn't mean that modeling has no place in financial planning. It does mean that model results should not be interpreted in a black-and-white context, but instead should be considered in shades of grey and within the limitations of the model and inputs.

An appropriate use of a good model in financial planning might be as an additional check, to suggest some aspects of a finished and supposedly final plan that may benefit from further scrutiny. An inappropriate usage would be to tell the client that his plan is a "GO" simply because the model says so.

Even our beloved Firecalc should not be relied upon as a sole basis for going ahead with a retirement plan. Still, it is not without its uses.

"All models are wrong. Some models are useful." - - George E.P. Box
__________________
Already we are boldly launched upon the deep; but soon we shall be lost in its unshored, harbourless immensities.

- - H. Melville, 1851
W2R is online now   Reply With Quote
Old 05-03-2009, 08:51 AM   #9
Thinks s/he gets paid by the post
 
Join Date: Jul 2004
Posts: 1,072
"Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it."

The most important sentence in the article - hello! Anybody who's had any math/engineering background know the assumptions make/break you. Expecting a model with assumptions to get even close to real life close to 100% of the time means one is quite naive. Bernstein's right - add a little extra padding - for him it's 20%......

Just because you've run the equation/model millions of times doesn't mean it predicts the true outcome - gotta have the right equation......GIGO
__________________
Deserat aka Bridget
“We sleep soundly in our beds because rough men stand ready in the night to visit violence on those who would do us harm.” - George Orwell/Winston Churchill
deserat is offline   Reply With Quote
Old 05-03-2009, 08:57 AM   #10
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
 
Join Date: Jul 2003
Location: Kansas City
Posts: 7,408
I believed. I believed. The Slosh model said the levees could/would go at Cat. 3. When you lived outside the levee over Lake Ponchartrain - you boogied.

Now on a hill, a BIG hill above the Missouri across from flat Kansas have a solid house WITH basement - near the tornado siren.

I like playing with models - but the Norwegian widow watches SEC yield like a hawk.

And I periodically - non scientifically cross check pssst-you know who top ten stock holdings and SEC yield to see how my portfolio compares.

heh heh heh -
__________________
unclemick is offline   Reply With Quote
Old 05-03-2009, 09:05 AM   #11
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
Midpack's Avatar
 
Join Date: Jan 2008
Location: Chicagoland
Posts: 11,973
Quote:
Originally Posted by deserat View Post
"Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it."

The most important sentence in the article - hello! Anybody who's had any math/engineering background know the assumptions make/break you. Expecting a model with assumptions to get even close to real life close to 100% of the time means one is quite naive. Bernstein's right - add a little extra padding - for him it's 20%......

Just because you've run the equation/model millions of times doesn't mean it predicts the true outcome - gotta have the right equation......GIGO
Life is uncertain, it's curious that anyone assumes otherwise in retirement (see virtually all online retirement calculators). Financial plans are no different, and no amount of modeling can change that. However, adding MC is still far, far better than the constant return/inflation & fixed longevity models that many people use if for no other purpose than to imprint the concept of uncertainty and planning for same for those who haven't thought through it. I have read this The Retirement Calculator from Hell, Part III several times, and keep it in mind always. I think Dr. Bernstein's whole 5-part Retirement Calculator From Hell series is worthwhile reading for everyone planning their own retirement (the non-SIRE crowd). 95% probability is more secure than 80% - it's just not actually 95%...
Quote:
A wildly optimistic historian might give us another few centuries of economic, political, and military continuity. Back-of-the-envelope, that’s about an 80% survival rate over the next 40 years. Thus, any estimate of long-term financial success greater than about 80% is meaningless.
__________________
No one agrees with other people's opinions; they merely agree with their own opinions -- expressed by somebody else. Sydney Tremayne
Retired Jun 2011 at age 57

Target AA: 60% equity funds / 35% bond funds / 5% cash
Target WR: Approx 2.5% Approx 20% SI (secure income, SS only)
Midpack is offline   Reply With Quote
Old 05-03-2009, 10:29 AM   #12
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
 
Join Date: May 2004
Posts: 11,615
Quote:
Originally Posted by mark View Post
It also says that by using tens or hundreds of thousands of iterations, the accuracy is improved.
FIRECalc doesn't use that many interations and does not incorporate "fatter tailed distributions".
I think the article overstates the significance of adding more simulation runs. That's not an important source of the expected vs real-world discrepency we are all seeing today. If the assumed variability in asset class returns represented what we have seen in the real world, and if the correlations between asset class returns were the same as what have actually occurred, then it wouldn't matter much if you ran 5,000 iterations or a million, the failures would show up.

FIRECalc ain't perfect, but it does at least include the worst-seen-to-date data. As the data from the present market stumble gets included, you'll have some more fatness in the tails. And, at that point, the relatively limited number of data years in FIRECalc will be an asset, serving to assure the "stinkers" get appropriate weight. This isn't a once-in-a-million event we're experiencing now.
__________________
"Freedom begins when you tell Mrs. Grundy to go fly a kite." - R. Heinlein
samclem is online now   Reply With Quote
Old 05-03-2009, 12:51 PM   #13
Moderator Emeritus
Nords's Avatar
 
Join Date: Dec 2002
Location: Oahu
Posts: 26,617
Quote:
Originally Posted by samclem View Post
This isn't a once-in-a-million event we're experiencing now.
I could live with once every decade or two...
__________________
*
*

The book written on E-R.org, "The Military Guide to Financial Independence and Retirement", on sale now! For more info see "About Me" in my profile.
I don't spend much time here anymore, so please send me a PM. Thanks.
Nords is offline   Reply With Quote
Old 05-03-2009, 01:25 PM   #14
Thinks s/he gets paid by the post
 
Join Date: Oct 2006
Posts: 3,820
Sam questions this statement. I do, too:

Quote:
Some industry participants also are trying to set standards that could help Monte Carlo tools more accurately capture extreme market events. The Retirement Income Industry Association in 2007 issued a set of principles noting that the calculators should run a large number of scenarios.
The ideal models run tens of thousands or hundreds of thousands of scenarios, which help gauge extreme events at the tail end of the distribution, observers said. Yet some tools run only 1,000 scenarios or just several hundred.
For retirement planning, lots of people accept 95% probability from the model. AFAIK, for a fixed set of assumptions, the 95% SWR is effectively the same whether you run 1,000 or 100,000 scenarios.

You're better off thinking about things outside the model then worrying about running a lot more scenarios.
__________________
Independent is offline   Reply With Quote
Old 05-03-2009, 01:36 PM   #15
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
Gone4Good's Avatar
 
Join Date: Sep 2005
Posts: 5,381
Quote:
Originally Posted by brewer12345 View Post
100% doesn't mean you are safe, either. The future may (quite likely, IMO) contain something worse than the historical record.
In a sense it already has. Using Shiller's data on stock returns and inflation, it looks to me like someone retiring at the 2000 peak has fared worse so far than someone who retired at the peak of 1929. That's in large part because deflation helped the 1929 retiree stretch his depleted portfolio further.
__________________
Gone4Good is offline   Reply With Quote
Old 05-03-2009, 01:52 PM   #16
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
Gone4Good's Avatar
 
Join Date: Sep 2005
Posts: 5,381
Quote:
A wildly optimistic historian might give us another few centuries of economic, political, and military continuity. Back-of-the-envelope, that’s about an 80% survival rate over the next 40 years. Thus, any estimate of long-term financial success greater than about 80% is meaningless.
I agree with Bernstein, I guess, but does this really have any practical value?

Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rational response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.
__________________
Gone4Good is offline   Reply With Quote
Old 05-03-2009, 01:58 PM   #17
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
Midpack's Avatar
 
Join Date: Jan 2008
Location: Chicagoland
Posts: 11,973
Quote:
Originally Posted by . . . Yrs to Go View Post
I agree with Bernstein, I guess, but does this really have any practical value?

Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rationale response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.
As I said in my earlier post, I think the value is in realizing "95% [model] probability is [definitely] more secure than 80% - it's just not actually 95%..." and planning accordingly. High model probabilities give a false sense of security which could lead to a catastrophic 'end of plan.' And it's just as likely any of us could expire ahead of schedule and deny ourselves unnecesarily. All of the models have a value as long as you understand them. So many people seem to search for retirement model nirvana, and there is no such thing, there are too many variables and too many unknowns --- period.

Just as we have to adapt during our working lives, we have to be prepared to adapt during our retired lives - until we expire.
__________________
No one agrees with other people's opinions; they merely agree with their own opinions -- expressed by somebody else. Sydney Tremayne
Retired Jun 2011 at age 57

Target AA: 60% equity funds / 35% bond funds / 5% cash
Target WR: Approx 2.5% Approx 20% SI (secure income, SS only)
Midpack is offline   Reply With Quote
Old 05-03-2009, 02:38 PM   #18
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
brewer12345's Avatar
 
Join Date: Mar 2003
Posts: 16,391
Quote:
Originally Posted by . . . Yrs to Go View Post
I agree with Bernstein, I guess, but does this really have any practical value?

Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rationale response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.
I gotta agree. Hedge all you like and save as much as you can, but at some point you pays your money and you takes your chances.

I will say that all of this makes me a lot more interested in a COLA'd pension or similar that is safe. It would at least put a floor under the worst outcomes. But I don't know where one would get one of these aside from SS and/or a federal pension.
__________________
"There are three kinds of men. The one that learns by reading. The few who learn by observation. The rest have to pee on the electric fence for themselves."



- Will Rogers
brewer12345 is offline   Reply With Quote
Old 05-03-2009, 02:39 PM   #19
Give me a museum and I'll fill it. (Picasso)
Give me a forum ...
 
Join Date: Jul 2003
Location: Kansas City
Posts: 7,408
Quote:
Originally Posted by . . . Yrs to Go View Post
I agree with Bernstein, I guess, but does this really have any practical value?

Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rationale response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.
Angus Maddison. .

A valid passport and do not be on the last train leaving Berlin.

heh heh heh - and pick your next destination well. 'Rich as an Argentine' is not just a Bernstein joke. . Hopefully not a problem in my remaining lifetime.
__________________
unclemick is offline   Reply With Quote
Old 05-03-2009, 04:32 PM   #20
Thinks s/he gets paid by the post
 
Join Date: Feb 2007
Posts: 1,905
Quote:
Originally Posted by samclem View Post
I think the article overstates the significance of adding more simulation runs. That's not an important source of the expected vs real-world discrepency we are all seeing today. If the assumed variability in asset class returns represented what we have seen in the real world, and if the correlations between asset class returns were the same as what have actually occurred, then it wouldn't matter much if you ran 5,000 iterations or a million, the failures would show up.

FIRECalc ain't perfect, but it does at least include the worst-seen-to-date data. As the data from the present market stumble gets included, you'll have some more fatness in the tails. And, at that point, the relatively limited number of data years in FIRECalc will be an asset, serving to assure the "stinkers" get appropriate weight. This isn't a once-in-a-million event we're experiencing now.
Agreed. An additional point to consider re FIRECalc is that the actual investment return data for the last century includes the time period when the US turned into the preeminent world power with an increase in wealth and productivity that may be difficult to match going forward. I wonder what the results of a FIRECalc type analysis would be if instead of using US data one were to use say Germany's or Japan or heaven forbid Argentina!
__________________

__________________
ejman is offline   Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Isn't FireCalc and Monte Carlo simulations based on higher gains? modhatter FIRE and Money 17 10-02-2006 04:32 AM
Monte Carlo al4trade FIRE and Money 1 08-13-2006 05:56 PM
Monte Carlo question modlair FIRECalc support 2 05-26-2006 07:51 PM
monte carlo Mysto FIRE and Money 5 04-18-2006 07:34 PM
Monte Carlo now can be used by brokers newellcr FIRE and Money 3 02-18-2005 09:41 AM

 

 
All times are GMT -6. The time now is 01:33 PM.
 
Powered by vBulletin® Version 3.8.8 Beta 1
Copyright ©2000 - 2017, vBulletin Solutions, Inc.