Monte Carlo Simulations-now recognized as understating risk

[samclem]

The WSJ had a piece this weekend on Monte Carlo simulations, and the fact that now (surprise!!) many people believe the simulations understate the risks of real world investing.

Some portions:

If one had asked a financial adviser 18 months ago for retirement-planning guidance, there is a good chance he would have run a "Monte Carlo" simulation. . .
But there is little chance your Monte Carlo simulation would have highlighted a scenario like the market slide just seen. Though these tools typically run a portfolio through hundreds or thousands of potential market scenarios, they often assign minuscule odds to extreme market events. Yet these extreme events seem to be happening more often.
. . .
Monte Carlo simulation has wide appeal, and is used in online tools offered by firms like Fidelity Investments and by independent retirement planners.
. . .

Many providers of the tools argue that it is a significant improvement over the traditional retirement-planning approach, which typically involves assuming some set market return, say 8% for U.S. stocks, year after year, an assumption considered unrealistic by academics and financial pros. The questions about Monte Carlo tools reflect broader concerns about mathematical models for gauging portfolio risks. . . .
These models were supposed to help quantify and manage the risks of mortgage-backed securities, credit-default swaps and other complex instruments. But given the events of the past couple of years, it appears that the models often gave big institutions, as well as small investors, a false sense of security.
. . . .
Now, some investors have decided that if risk can't be accurately measured, they will just have to play it safe. . . .
Some financial advisers are equally skeptical. "I take whatever probability of failure that comes out of your Monte Carlo simulation and add 20 percentage points," said William J. Bernstein, author of "The Four Pillars of Investing."
Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it. Since no standard approach exists, one user might plug in a range of assumptions on interest rates, inflation or volatility that is different from another user.
Also controversial is that many Monte Carlo simulations assume that market returns fall along a bell-curve-shaped distribution. . . .
"In a bell-shaped curve the probability of getting one of these extreme outcomes we're seeing is basically zero," said Paul Kaplan, vice president of quantitative research at Morningstar Inc.
. . . .
While a bell-curve model indicates there is almost no chance of a greater than 13% monthly decline in the Standard & Poor's 500-stock index, such declines have happened at least 10 times since 1926, according to a report by Mr. Kaplan.
Some Monte Carlo models, like the one used by Financial Engines, assign higher odds to extreme market events than the bell-curve distributions. . . .

Some firms are considering revising Monte Carlo models to reflect a world where big market swings happen more often. Morningstar last year tweaked its asset-allocation software offered to institutional investors, allowing users to choose a bell-curve-shaped distribution or a "fat-tailed" distribution, which assigns higher probabilities to extreme market events. The company is exploring using this model in more products, Mr. Kaplan said.
Laurence Kotlikoff, a Boston University economics professor who developed the ESPlanner financial-planning software, and Richard Fullmer, senior portfolio strategist at Russell Investments, said they also are considering offering clients Monte Carlo scenarios that incorporate fatter-tailed distributions. . . .

Some industry participants also are trying to set standards that could help Monte Carlo tools more accurately capture extreme market events. The Retirement Income Industry Association in 2007 issued a set of principles noting that the calculators should run a large number of scenarios.
The ideal models run tens of thousands or hundreds of thousands of scenarios, which help gauge extreme events at the tail end of the distribution, observers said. Yet some tools run only 1,000 scenarios or just several hundred.

It's good that MC simulations are getting more attention. Still, this article didn't do a good job of explaining some of the common weaknesses (esp capturing complex interdependencies among asset classes and adjacent-year returns, and instead treating them as independent. This significantly understates risk).

As Bernstein has pointed out in the past, there's systemic risk (including the fairly regular collapse of governments and economic systems) that overwhelms some of the miniscule failure rates predicted by these MC simulators.

Also, too bad they didn't touch on FIRECalc.

 

[Nords]

But... but... but... this would mean that returns aren't log-normal distributions!

Apparently quite a bit has been lost in translation between the research papers and the mutual-fund company websites.

 

[Maurice]

This is Nicholas Nassim Taleb's 'black swan' point exactly, applied to MC analyses.

As for how important it is, it depends on how the MC analysis is being used. In many banks, MC analyses are used to value many derivatives with path-dependent valuations. In this case using a gaussian distribution is probably folly.

But in many fire-calc type analyses you use the gaussian distribution but then set such a high bar on your planning (say 95% survival rate) that you're really giving much more weight to bad events than is implied through the use of the gaussian model.

 

[rjsciurus]

I always wondered about the logic of using something like the 95% success rate. That means that in the last 70-100 years (depending on your data set), your worst case scenario did happen. Considering the down side-- having no personal resources in your helpless advanced old age-- that doesn't seem very prudent. Picking 100% at least attempts to factor in *all* catastrophic economic events within living memory. I'm a Taleb fan, obviously.

 

[brewer12345]

I always wondered about the logic of using something like the 95% success rate. That means that in the last 70-100 years (depending on your data set), your worst case scenario did happen. Considering the down side-- having no personal resources in your helpless advanced old age-- that doesn't seem very prudent. Picking 100% at least attempts to factor in *all* catastrophic economic events within living memory. I'm a Taleb fan, obviously.

100% doesn't mean you are safe, either. The future may (quite likely, IMO) contain something worse than the historical record.

You pays your money and you takes your chances, same as always, Taleb's obnoxious commercializing of this central concept notwithstanding.

 

[Maurice]

Yeah, I like the analyses that show actuals from the late 1800s on, its a nice compliment to MC analyses.

On the other hand, most of these models don't take into account the fact that our reactions to changing markets are dynamic. Most of us don't continue taking out the same amount in bad years - we cut back and take less if we can.

 

[mark]

It also says that by using tens or hundreds of thousands of iterations, the accuracy is improved.
FIRECalc doesn't use that many interations and does not incorporate "fatter tailed distributions".

 

[W2R]

Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it. Since no standard approach exists, one user might plug in a range of assumptions on interest rates, inflation or volatility that is different from another user.
Also controversial is that many Monte Carlo simulations assume that market returns fall along a bell-curve-shaped distribution. . . .

Modeling is fun, and people without sufficient modeling background frequently use (mis-use) models as described. This is nothing new. Poor problem selection is an egregious mis-use of models. Poor interpretation of results that were obtained based on poor problem selection and inappropriate assumptions is simply ignorant, IMO.

Financial planning problems involve a huge numbers of unknowns and assumptions and also unknown probabilities ("past performance is no guarantee of future results" points out just one of many possible bad assumptions). That doesn't mean that modeling has no place in financial planning. It does mean that model results should not be interpreted in a black-and-white context, but instead should be considered in shades of grey and within the limitations of the model and inputs.

An appropriate use of a good model in financial planning might be as an additional check, to suggest some aspects of a finished and supposedly final plan that may benefit from further scrutiny. An inappropriate usage would be to tell the client that his plan is a "GO" simply because the model says so.

Even our beloved Firecalc should not be relied upon as a sole basis for going ahead with a retirement plan. Still, it is not without its uses.

"All models are wrong. Some models are useful." - - George E.P. Box

 

[deserat]

"Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it."

The most important sentence in the article - hello! Anybody who's had any math/engineering background know the assumptions make/break you. Expecting a model with assumptions to get even close to real life close to 100% of the time means one is quite naive. Bernstein's right - add a little extra padding - for him it's 20%......

Just because you've run the equation/model millions of times doesn't mean it predicts the true outcome - gotta have the right equation......GIGO

 

[unclemick]

I believed. I believed. The Slosh model said the levees could/would go at Cat. 3. When you lived outside the levee over Lake Ponchartrain - you boogied.

Now on a hill, a BIG hill above the Missouri across from flat Kansas have a solid house WITH basement - near the tornado siren.

I like playing with models - but the Norwegian widow watches SEC yield like a hawk.

And I periodically - non scientifically cross check pssst-you know who top ten stock holdings and SEC yield to see how my portfolio compares.

heh heh heh -

 

[Midpack]

"Critics emphasize that the problem isn't Monte Carlo itself, but the assumptions that go into it."

The most important sentence in the article - hello! Anybody who's had any math/engineering background know the assumptions make/break you. Expecting a model with assumptions to get even close to real life close to 100% of the time means one is quite naive. Bernstein's right - add a little extra padding - for him it's 20%......

Just because you've run the equation/model millions of times doesn't mean it predicts the true outcome - gotta have the right equation......GIGO

Life is uncertain, it's curious that anyone assumes otherwise in retirement (see virtually all online retirement calculators). Financial plans are no different, and no amount of modeling can change that. However, adding MC is still far, far better than the constant return/inflation & fixed longevity models that many people use if for no other purpose than to imprint the concept of uncertainty and planning for same for those who haven't thought through it. I have read this The Retirement Calculator from Hell, Part III several times, and keep it in mind always. I think Dr. Bernstein's whole 5-part Retirement Calculator From Hell series is worthwhile reading for everyone planning their own retirement (the non-SIRE crowd). 95% probability is more secure than 80% - it's just not actually 95%...

A wildly optimistic historian might give us another few centuries of economic, political, and military continuity. Back-of-the-envelope, that’s about an 80% survival rate over the next 40 years. Thus, any estimate of long-term financial success greater than about 80% is meaningless.

 

[samclem]

It also says that by using tens or hundreds of thousands of iterations, the accuracy is improved.
FIRECalc doesn't use that many interations and does not incorporate "fatter tailed distributions".

I think the article overstates the significance of adding more simulation runs. That's not an important source of the expected vs real-world discrepency we are all seeing today. If the assumed variability in asset class returns represented what we have seen in the real world, and if the correlations between asset class returns were the same as what have actually occurred, then it wouldn't matter much if you ran 5,000 iterations or a million, the failures would show up.

FIRECalc ain't perfect, but it does at least include the worst-seen-to-date data. As the data from the present market stumble gets included, you'll have some more fatness in the tails. And, at that point, the relatively limited number of data years in FIRECalc will be an asset, serving to assure the "stinkers" get appropriate weight. This isn't a once-in-a-million event we're experiencing now.

 

[Nords]

This isn't a once-in-a-million event we're experiencing now.

I could live with once every decade or two...

 

[Independent]

Sam questions this statement. I do, too:

Some industry participants also are trying to set standards that could help Monte Carlo tools more accurately capture extreme market events. The Retirement Income Industry Association in 2007 issued a set of principles noting that the calculators should run a large number of scenarios.
The ideal models run tens of thousands or hundreds of thousands of scenarios, which help gauge extreme events at the tail end of the distribution, observers said. Yet some tools run only 1,000 scenarios or just several hundred.

For retirement planning, lots of people accept 95% probability from the model. AFAIK, for a fixed set of assumptions, the 95% SWR is effectively the same whether you run 1,000 or 100,000 scenarios.

You're better off thinking about things outside the model then worrying about running a lot more scenarios.

 

[Gone4Good]

100% doesn't mean you are safe, either. The future may (quite likely, IMO) contain something worse than the historical record.

In a sense it already has. Using Shiller's data on stock returns and inflation, it looks to me like someone retiring at the 2000 peak has fared worse so far than someone who retired at the peak of 1929. That's in large part because deflation helped the 1929 retiree stretch his depleted portfolio further.

 

[Gone4Good]

A wildly optimistic historian might give us another few centuries of economic, political, and military continuity. Back-of-the-envelope, that’s about an 80% survival rate over the next 40 years. Thus, any estimate of long-term financial success greater than about 80% is meaningless.

I agree with Bernstein, I guess, but does this really have any practical value?

Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rational response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.

 

[Midpack]

I agree with Bernstein, I guess, but does this really have any practical value?

Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rationale response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.

As I said in my earlier post, I think the value is in realizing "95% [model] probability is [definitely] more secure than 80% - it's just not actually 95%..." and planning accordingly. High model probabilities give a false sense of security which could lead to a catastrophic 'end of plan.' And it's just as likely any of us could expire ahead of schedule and deny ourselves unnecesarily. All of the models have a value as long as you understand them. So many people seem to search for retirement model nirvana, and there is no such thing, there are too many variables and too many unknowns --- period.

Just as we have to adapt during our working lives, we have to be prepared to adapt during our retired lives - until we expire.

 

[brewer12345]

I agree with Bernstein, I guess, but does this really have any practical value?

Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rationale response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.

I gotta agree. Hedge all you like and save as much as you can, but at some point you pays your money and you takes your chances.

I will say that all of this makes me a lot more interested in a COLA'd pension or similar that is safe. It would at least put a floor under the worst outcomes. But I don't know where one would get one of these aside from SS and/or a federal pension.

 

[unclemick]

I agree with Bernstein, I guess, but does this really have any practical value?

Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rationale response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.

Angus Maddison. .

A valid passport and do not be on the last train leaving Berlin.

heh heh heh - and pick your next destination well. 'Rich as an Argentine' is not just a Bernstein joke. . Hopefully not a problem in my remaining lifetime.

 

[ejman]

I think the article overstates the significance of adding more simulation runs. That's not an important source of the expected vs real-world discrepency we are all seeing today. If the assumed variability in asset class returns represented what we have seen in the real world, and if the correlations between asset class returns were the same as what have actually occurred, then it wouldn't matter much if you ran 5,000 iterations or a million, the failures would show up.

FIRECalc ain't perfect, but it does at least include the worst-seen-to-date data. As the data from the present market stumble gets included, you'll have some more fatness in the tails. And, at that point, the relatively limited number of data years in FIRECalc will be an asset, serving to assure the "stinkers" get appropriate weight. This isn't a once-in-a-million event we're experiencing now.

Agreed. An additional point to consider re FIRECalc is that the actual investment return data for the last century includes the time period when the US turned into the preeminent world power with an increase in wealth and productivity that may be difficult to match going forward. I wonder what the results of a FIRECalc type analysis would be if instead of using US data one were to use say Germany's or Japan or heaven forbid Argentina!

 

[Dennis]

Well, I looked at alot of these Self Proclaimed Guru's Plans and methods and systems, prior to about 5 yrs before retiring... and then I just used my own Simple Minded Common sense...

Example: Bernstiens Retirement from Hell..."To make $100k yr at a 4% SWR you neng ave of $1. 769k?"
I don't know where he gets his #'s, but I come up with $250k and After paying 25% in Fed and state taxes, I'd have 3% net to spend. and adding another 4% or 3% net after taxes left in for inflation/buying power.. tells me I need a min. 8% apy on my $. and it's reasonable to make 10% apy on a 60/40 port? I don';t know about that either, a 60/40 port of indexes, Ending in 07' only had an ave of about 7.4% apy for the previous 10 yrs, add 08' and it's now the past 10 yrs ending 08' of only about 3.3% apy..

This , among other reasons moved me to get out of Indexes and get into Active Balanced Funds.. way back in 98/99', that had and still have the best chance of meeting and succeeding that 8% apy I'm looking for..and even after 08', the 4 BF's I have are still in the 9% apy range for the past 9 yrs now.. and I added an extra 20% to what everyone and Myself figured I would need to have ...thus worked an extra 4 yrs...

and being a person who believes you need a Speicalist for everything Now-a-days and Most of Us Amatures Haven't Got a Chance doing out own Investing..and we Need a Pro' to Beat a Pro'...and one with a proven record and not therories or Charts...with Real $..( most Advisors Won't share what their Past 3,5 & 10 yr Recommended ports did to you )

and everytime ( about every 6mos) I go see one of these Financial Advisors or A Firm offerring to do a Port. review to get my $? After they give me their recomendations for a conservative to Moderate Port? I pull out mine and none have yet to get close, let alone beat them and they critize them and hate them...

Which tells me, I'm on the right course...LOL

But, Like the Guy who Jumped off a 40 story building , said at the 30th floor?
Well, So Far, So good...
;>)

 

[samclem]

Example: Bernstiens Retirement from Hell..."To make $100k yr at a 4% SWR you neng ave of $1. 769k?"

He didn't say that.
(From "Retirement Calculator From Hell-Part III")

For centuries, investors used the amortization algorithm—the same formula used to calculate mortgages. Let’s say that you plan a 30-year retirement, estimate a 4% real return, and need $100,000 in annual income. Toss these figures into your trusty retirement calculator, and hey presto, out pops a required nest egg of $1,729,203.

Note that the 4% is real return (i.e. after inflation), and that every last penny is spent by the end of 30 years. Also, of course, he goes on to say how silly these straight-line calculations are, proposes MC as something better, then notes the limitations of any quantitative modelling.

 

[clifp]

Modeling is fun, and people without sufficient modeling background frequently use (mis-use) models as described.

"All models are wrong. Some models are useful." - - George E.P. Box

I love this quote. Is there a detailed explanation of how FIRECalc does modeling. I know it uses historical data rather than a MC simulation.

What are the implications for this type of modeling vs MC?

 

[Nords]

I agree with Bernstein, I guess, but does this really have any practical value?
Is it worthwhile to factor in probable outcomes over which there is no possible hedge? Working 10 more years while saving twice as much does nothing to move the needle above 80% in Bernstein's view . . . so why bother? In fact, the only rational response to a 1 in 5 chance of losing everything is to hurry up and spend it before its gone. Saving more, meanwhile, makes very little sense.

This is why retirees buy annuities. Even Milevsky has come to favor them for most situations, which is a course reversal from his 1990s research & conclusions.

I love this quote. Is there a detailed explanation of how FIRECalc does modeling. I know it uses historical data rather than a MC simulation.
What are the implications for this type of modeling vs MC?

Better give yourself plenty of time and math when those physical oceanographers start talking numerical analysis & modeling simulation!

FIRECalc is only going on history, instead of potentially flawed assumptions, but it has to deal with a smaller number of runs or make approximations to deal with partial runs. For example in 125 years of data there will be 95 runs of 30-year retirements, but only 75 runs of 50-year retirements. Neither one may be enough to reach a rigorous confidence level.

FIRECalc will more accurately reflect correlation between returns (four good years in a row or two consecutive crappy years) where most MC simulations sequence them randomly. Recent (more sophisticated) MC software now tries to correlate returns instead of abruptly shifting from gains to losses.

FIRECalc's history will also more accurately reflect changing correlations between asset classes. Again MC is attempting to catch up with these new factors whenever they arise.

Of course Dory explains it much better than I do, and with pictures: FIRECalc: Why another retirement calculator?

 

[NW-Bound]

About modelling, in my work we use mathemetical modeling all the time. Though the analytical models are numerical and spit out results to many digits of precision, we use it more as a qualitative tool than a quantitative one. We can conduct experiments or try out many different setups or inputs more easily with a model than with real-life. This narrows down our choices to try later with the real experimental prototypes. The real test results are then analyzed and used to refine the model. After some iterations, we may be able to see if our model starts to resemble real life. In some cases, we were never able to get as close as we originally thought possible. Nature is much more complex than our equations, no matter how sophisticated the latter look on paper.

In economics, it is even tougher, as it is also not possible to conduct controlled experiments with all factors under control or at least tightly monitored. In my field, we have all these luxuries. Yet, we still often get that unexpected disappointment with objects that have to obey the laws of physics, not something that is at the whim of politicians or their short-attention-span constituents, wars, natural disasters, and swine flus.

Yet, we only have one portfolio to manage for our retirement. There is no second chance, no rerun.

Gerald Loeb in his book "The Battle for Investment Survival" observed that "number" men (whatever that means) made the worst investors, while psychologists would be better ones. I do not know if there is any statistics to back this up. However, it appears to be true among my work circle that engineers often express frustration with the market not "working out" to their expectation.

Agreed. An additional point to consider re FIRECalc is that the actual investment return data for the last century includes the time period when the US turned into the preeminent world power with an increase in wealth and productivity that may be difficult to match going forward. I wonder what the results of a FIRECalc type analysis would be if instead of using US data one were to use say Germany's or Japan or heaven forbid Argentina!

I think ejman hit it on the head! As it is not likely that this century will play out like the last one, it's better to hedge and look to invest in overseas economies, or at least US multinational corporations.