Monte Carlo or Historical in ***** and Firecalc?

[bulbar]

When I run my plan in ***** using the Historicals, I get 90%+ success rate. But then I switch it to Monte Carlo and I get 67%.

When you guys talk about your % success rates, is that using historicals or monte carlo? Should I be using Monte Carlo all the time? I've been using Historicals.

 

[REWahoo]

Historical.

 

[ERD50]

The default for FIRECalc is historical, I'd assume anyone doing Monte Carlo would specify that. IMO, almost everyone using those tools with historical data is using the historical mode - it is what they are designed to do, and makes them different (and far more valuable IMO), than the others.

With Monte Carlo, you never really know what you are getting. How much variation did the programmer put into it, did they try to match historical correlation of different investments and inflation? If so, why not just use historical.

Nothing can predict the future, but I'd rather base some numbers on what we know has happened, rather than some programmer's model. And they don't usually share their code, so how can you check it? At least with historical, you can make some checks - the data is public.

PS - in case you didn't catch my drift, I don't care for and don't use Monte Carlo - unless I'm doing some engineering analysis, it has a place in some cases, but not this one, IMO.

-ERD50

 

[MichaelB]

Historical for FIRECalc

 

[BBQ-Nut]

When I run my plan in ***** using the Historicals, I get 90%+ success rate. But then I switch it to Monte Carlo and I get 67%.

When you guys talk about your % success rates, is that using historicals or monte carlo? Should I be using Monte Carlo all the time? I've been using Historicals.

Both-just to get an idea of different results.

I also change some of the default values for return, std deviation, and inflation when running a MC scenario for "Portfolio with Random Performance" to bracket success rate.

Also, I've noticed that you get different MC success percentages running the exact same values due to the random number 'seed' that a given run may use. So, I make lots of MC runs to see what the spread of success may be.

 

[Sunset]

Oops, too late to do the Monte Carlo as we already retired

 

[photoguy]

MC can be useful IF you understand how the simulation works and the assumptions behind the model. It has the advantage of being able to run different scenarios (especially those that do not have good representation in the historical data)

But there are also a lot of drawbacks -- complex models are harder to understand, are not necessarily more accurate, easier to make implementation mistakes, and easier to fudge if one wants to push an agenda.


Sent from my iPad using Early Retirement Forum

 

[robertf57]

Given the stark differences, did you use a 40 year period of retirement or something? There are less 40 year periods in the historical database and they lead to a somewhat rosey projection because they under-represent some of the more challenging recent periods and over-represent some prosperous periods. I have used 30 year periods for my firecalc models...

 

[big-papa]

By the way, Jim Otar's book has a good section on Monte Carlo. Worth a read, if you have a math background. Even if you don't, he recognizes that returns (at least the SP500) goes through long-ish periods of nearly constant growth at a certain growth rate (with noise around that, of course). He has a MC simulator that takes that into account, even though he doesn't like MC at all. Most MC simulations assume each year is completely random with no correlation year-to-year. Will the future do likewise - who knows?

 

[pb4uski]

The difference in your outcomes seems odd. I tested with firecalc assuming $1m portfolio and $40k withdrawals over 30 years, 60/40 portfolio and all other assumptions default assumptions.

Base case: historical: 95.6%
Monte Carlo w/8.8% average return, default std dev*: 98.6%, 100% and 95.9%
Vanguard Monte Carlo simulator: 93%, 92%, 93%

* historical average for 60/40 portfolio per Vanguard

I ran the Monte Carlo three times for each tool.

 

[gcgang]

I use a MC program, believing it's preferable to look at 2000+ sets of returns rather than one historical.




Sent from my iPad using Early Retirement Forum

 

[nuke_diver]

I use both but I'm not a big fan of FireCalc (or ********)'s method of MC. I prefer Flexible Retirement planner for that since it is far more...flexible

Historical is nice but history may not repeat (the conditions that made those markets might not happen in the future I mean...like a couple of world wars and then booms after for example). The good and bad things of MC is that done correctly they will be totally random...so a decade or 2 of declines is possible (and the reverse). Not something that is necessarily likely but if you can survive that you are probably good to go. In FRP it shows the # of times the money ran out as well as the 10/90%ile levels so you can tell if the money ran out under a severe outlier condition or not.

I do get much better results from historical than from MC especially if I set up a scenario with a sharp decrease of say 30-50% (sort of comparing 1929 followed by MC rather than 1929 followed by the historical results).

 

[bulbar]

I tried Flexible Planner. It gives me 100% using a fixed 5.5% return, no st deviation. Then I used the monte carlo. I used the moderate risk assumptions and it gave me 85%. I tried the other risk categories and got about the same. Much higher than the 67% monte carlo I was getting with *****.

 

[pb4uski]

Wouldn't 5.5% fixed with no standard deviation be the same as 5.5% deterministic?

 

[samclem]

I use a MC program, believing it's preferable to look at 2000+ sets of returns rather than one historical.

2000 sets of notional returns or one set of real returns, right?

There's no perfect answer, and this has been studied a lot. There is definite correlation between years and between data sets. Some MC sims attempt to model these, some just go with random returns that fit into the defined SD window, but don't attempt to match the corelations of real life. To use data that assumes (for example) that inflation is not related to the returns on fixed income investments seems to be not very useful. And if the MC has been carefully designed to match real-world correlations--at some point it makes just as much sense to use the real world data raw.
Some models I've seen use real data from more than one country to help retain the correlations while allowing for a different set of meta-economics (i.e. not just/primarily 20th century US).

 

[Fred123]

2000 sets of notional returns or one set of real returns, right?

There's no perfect answer, and this has been studied a lot. There is definite correlation between years and between data sets. Some MC sims attempt to model these, some just go with random returns that fit into the defined SD window, but don't attempt to match the corelations of real life. To use data that assumes (for example) that inflation is not related to the returns on fixed income investments seems to be not very useful. And if the MC has been carefully designed to match real-world correlations--at some point it makes just as much sense to use the real world data raw.
Some models I've seen use real data from more than one country to help retain the correlations while allowing for a different set of meta-economics (i.e. not just/primarily 20th century US).

An endless source of debate here! There are a number of different ways of doing MC, and, IMHO, one of the best ways is with the bootstrap (discussed previously here); you don't have to assume a distribution. Although fixed income time series do show autocorrelation, this isn't true for annual stock returns. And there are methods (for instance, block bootstrap), that deal with this problem.

I'm not a big fan of non-MC Firecalc, and this is confirmed (at least for me) by the fact that MC tends to be more pessimistic. I'm also uncomfortable with the small number (about 4 or 5) of truly independent time series that FC uses to predict outcomes.

 

[ERD50]

...

I'm not a big fan of non-MC Firecalc, and this is confirmed (at least for me) by the fact that MC tends to be more pessimistic. ...

And a few minor tweaks of the programmer's MC terms and/or algorithms will change that. So when is it 'right'?

What if M-C was less pessimistic than history? Would it be 'wrong'? Sounds like you are still using history as a base-line. Why not use history, plus a comfortable safety factor? I think MC is just a fancy, convoluted, complicated, circular means to that end. 'But a computer came up with this answer!'

I'm also uncomfortable with the small number (about 4 or 5) of truly independent time series that FC uses to predict outcomes.

Agreed, but since we can't predict the future, I thinks it's all we got. At some point, most of us just need to go with our decision, or work until we die.

I feel reasonable, going with 100% and ~ 45 years (before you start dropping off bad cycles), knowing that my portfolio should survive the worst of anything we've seen in history, plus a little fudge factor for safety. Just because we may have some future cycles that are unrepresented in history does not mean they will be worse than the worst ever thrown at us. And if they do, I've got some buffer. and will need to adjust - probably like everyone else, regardless what tool they used to determine their CWR (Comfortable Withdraw Rate).

-ERD50

 

[Fred123]

And a few minor tweaks of the programmer's MC terms and/or algorithms will change that. So when is it 'right'?

What if M-C was less pessimistic than history? Would it be 'wrong'? Sounds like you are still using history as a base-line. Why not use history, plus a comfortable safety factor? I think MC is just a fancy, convoluted, complicated, circular means to that end. 'But a computer came up with this answer!'





Agreed, but since we can't predict the future, I thinks it's all we got. At some point, most of us just need to go with our decision, or work until we die.

I feel reasonable, going with 100% and ~ 45 years (before you start dropping off bad cycles), knowing that my portfolio should survive the worst of anything we've seen in history, plus a little fudge factor for safety. Just because we may have some future cycles that are unrepresented in history does not mean they will be worse than the worst ever thrown at us. And if they do, I've got some buffer. and will need to adjust - probably like everyone else, regardless what tool they used to determine their CWR (Comfortable Withdraw Rate).

-ERD50

So, how exactly do you choose that fudge factor? The problem is that if you have no estimate of the variance of the distribution, then it's impossible to make any kind of intelligent guess as to what might work. The beauty of MC is that it provides a method for estimating the variance (or randomness if you like) in the distribution and gives you an idea of what that fudge factor should be!

And, really, there's nothing at all complicated about MC. In fact, it couldn't be simple. In the case of bootstrapping, if you're interested in a 30 year retirement period, you just select 30 random years from your historical financial time series and calculate the outcome in the same way as FC. Repeat this a 1000 times and you have your estimate.

 

[ERD50]

RE: fudge factor on historical...

So, how exactly do you choose that fudge factor? The problem is that if you have no estimate of the variance of the distribution, then it's impossible to make any kind of intelligent guess as to what might work. The beauty of MC is that it provides a method for estimating the variance (or randomness if you like) in the distribution and gives you an idea of what that fudge factor should be! ...

That would be easier for me to consider if there was some absolute definition for the M-C factors for an application such as this. But, AFAIK, it is up to the programmer to determine whether they are going to make any adjustments for correlations between different inputs (inflation versus interest rates, etc).

And, really, there's nothing at all complicated about MC. In fact, it couldn't be simple. In the case of bootstrapping, if you're interested in a 30 year retirement period, you just select 30 random years from your historical financial time series and calculate the outcome in the same way as FC. Repeat this a 1000 times and you have your estimate.

And there is the crux of the problem - I don't believe that financial markets are that simple. Yes, there is randomness year-to-year, but booms follow busts, and busts follow booms, with some flat levels in between, sometimes. I think there are some complex reasons for that, including human emotion and the 'madness of crowds'. The inflection points change, and the time and flat periods vary (they do this just to foul up any dirty-market-timer attempts! .

-ERD50

 

[galeno]

For our 60/40 port with a TER = 0.9%.


Historical Firecalc gives us an AWR = 3.6% for an 85% success rate.


Monte Carlo ORP gives us a 3.5% AWR.

 

[photoguy]

Although fixed income time series do show autocorrelation, this isn't true for annual stock returns. And there are methods (for instance, block bootstrap), that deal with this problem.

This is a really interesting point. I want to note that criticizing an MC method for not taking into account temporal correlations in returns is equivalent to suggesting that market timing works.

As a side note, block bootstrap probably won't work well in this case because we are interested in worst case/extreme outcomes and the bootstrap is not good for that type of statistic.

And, really, there's nothing at all complicated about MC. In fact, it couldn't be simple.

There's a huge range of MC models for retirement planning. Some of them may be very simple as you suggest, but others can be extremely complicated.

 

[ERD50]

... I want to note that criticizing an MC method for not taking into account temporal correlations in returns is equivalent to suggesting that market timing works. ...

I'll disagree - to a degree.

From what I've seen, there is a difference between recognizing that there are boom-bust cycles, and even being able to identify relatively high/low valuations in stocks - and being able to identify the profitable points to get in/out.

Look at the 1990's - I recall feeling very strongly that by ~ 1997, the market was 'hot' and would not go much higher, and I was not alone, not by a long shot. But if you got out, you would have missed much of the run, and to get back in you'd need to time it almost perfectly. Actually (I'd need to check the data more closely), considering you would make a few % in fixed income while sitting out, I don't think you could find a profitable entry point. That's where market timing fails, but isn't in denial of boom/bust and high/low valuations.


-ERD50

 

[Fred123]

I'll disagree - to a degree.

From what I've seen, there is a difference between recognizing that there are boom-bust cycles, and even being able to identify relatively high/low valuations in stocks - and being able to identify the profitable points to get in/out.

Look at the 1990's - I recall feeling very strongly that by ~ 1997, the market was 'hot' and would not go much higher, and I was not alone, not by a long shot. But if you got out, you would have missed much of the run, and to get back in you'd need to time it almost perfectly. Actually (I'd need to check the data more closely), considering you would make a few % in fixed income while sitting out, I don't think you could find a profitable entry point. That's where market timing fails, but isn't in denial of boom/bust and high/low valuations.


-ERD50

But isn't that the point? You can't predict what's going to happen at time t+1 at time t. Which, of course, is the definition of randomness or noise. Or no autocorrelation. Another way to look at it would be to consider what you call boom/bust cycles as random shocks -- the shocks change the market, but they're still unpredictable and as such need to be considered as noise.

 

[ERD50]

But isn't that the point? You can't predict what's going to happen at time t+1 at time t. Which, of course, is the definition of randomness or noise. Or no autocorrelation. Another way to look at it would be to consider what you call boom/bust cycles as random shocks -- the shocks change the market, but they're still unpredictable and as such need to be considered as noise.

An analogy - we have seasons here, Winter (and winter and winter and winter it seems like lately), Spring, Summer, and Fall.

We know that Spring and Summer follow Winter, and it will be warmer. But we can't say exactly when and by how much on any specific day/week. And after a warm spell, it might get cold again before it gets warmer. There is randomness, but there are still cycles.

What would be a better way to plan for a specific week of the year, a historical analysis of the limited data we have for temps and precipitation for that week, or a random mix of temperatures and precipitation for the year?

Of course, market cycles are not as predictable as the seasons, it's an analogy. But as an analogy, I think it has some value.

-ERD50

 

[photoguy]

Look at the 1990's - I recall feeling very strongly that by ~ 1997, the market was 'hot' and would not go much higher, and I was not alone, not by a long shot. But if you got out, you would have missed much of the run, and to get back in you'd need to time it almost perfectly. Actually (I'd need to check the data more closely), considering you would make a few % in fixed income while sitting out, I don't think you could find a profitable entry point. That's where market timing fails, but isn't in denial of boom/bust and high/low valuations.

I agree with Fred123 here. The reason the timing is so difficult is because the correlation (which is predictability under a linear model) is either zero or very low. If it were more predictable, the timing would be easier.

We know that Spring and Summer follow Winter, and it will be warmer. But we can't say exactly when and by how much on any specific day/week. And after a warm spell, it might get cold again before it gets warmer. There is randomness, but there are still cycles.

One doesn't need correlation to explain boom/bust cycles in stock returns. Even with a completely random process, you can get runs of up or down years which "explain" the cycles in data.