using actual data or monte carlo sims-4% rule

[ERD50]

This topic has come up before, and I'm always surprised at the aversion some people here have to MC analysis and the overconfidence they have in FC (which relies on perhaps 5 independent samples if you are looking at 30 year outcomes). Some of this seems to be based on a misunderstanding on how MC works if historical data is used to model the distribution. For example, there aren't "a s***load of parameters" used, there are two (mean and variance) for normal or log-normal distributions and maybe one or two others if a "fat-tailed" distribution is used. There are no parameters used at all if bootstrap sampling is used.

But, more importantly, what are the alternatives? As a famous statistician once said: "all models are wrong, but some are useful."

Who says anyone has 'overconfidence in FC'? This was recently covered in the 'pet peeve' thread - please don't put words in other posters's posts.

From my perspective, FC reports history. I am reasonably confident that it does this reasonably accurately (there are some questions about the bond re-balancing algorithm). What you decide to do based on that report is a personal choice. But I don't think it infers an 'overconfidence' in it. It shows ~ 100 different lines - which one am I supposedly 'confident' in? And I understand the future could be worse than the worst of the past. But I think it's a reasonable data-point.

You say that MC uses only two parameters? But on how many variables? FC is looking at market returns, fixed returns, and inflation. I don't believe those are totally independent, so how does MC handle any correlations (positive or negative)? Without a full understanding of that, I don't know what to think about any MC run.

FWIW, I approach the historical calculators conservatively. I plan for the outside chance one of us lives to 100 (beyond that is diminishing changes anyhow). I choose 100% success rates. Then I throw in a little buffer. How does that make me 'overconfident'?


edit/add: I agree with you that there are really only a few economic cycles represented in this history. But what else can we do? You need to start somewhere.

-ERD50

 

[M Paquette]

I love Monte Carlo simulations. Especially when I can buttonhole the designer and ask questions.

"I see you used a Gaussian random number generator. Does the output actually resemble the market behavior of that asset? How much divergence? I don't see where you allow for cross-correlated behavior for these different asset types? Does the histogram of first and second derivatives with respect to time resemble the behavior of the asset in the real world? How closely?"

The answers usually resemble the series "Um. Uh... I don't know. What? The who? Uh..."

 

[Fred123]

Who says anyone has 'overconfidence in FC'? This was recently covered in the 'pet peeve' thread - please don't put words in other posters's posts.

From my perspective, FC reports history. I am reasonably confident that it does this reasonably accurately (there are some questions about the bond re-balancing algorithm). What you decide to do based on that report is a personal choice. But I don't think it infers an 'overconfidence' in it. It shows ~ 100 different lines - which one am I supposedly 'confident' in? And I understand the future could be worse than the worst of the past. But I think it's a reasonable data-point.

You say that MC uses only two parameters? But on how many variables? FC is looking at market returns, fixed returns, and inflation. I don't believe those are totally independent, so how does MC handle any correlations (positive or negative)? Without a full understanding of that, I don't know what to think about any MC run.

FWIW, I approach the historical calculators conservatively. I plan for the outside chance one of us lives to 100 (beyond that is diminishing changes anyhow). I choose 100% success rates. Then I throw in a little buffer. How does that make me 'overconfident'?


edit/add: I agree with you that there are really only a few economic cycles represented in this history. But what else can we do? You need to start somewhere.

-ERD50

FWIW, I do think many posters here are overconfident about Firecalc. I'm not sure why you interpreted that as putting words in other's posts -- it's just my personal opinion and I expressed it as such. I do seem to read quite a few posts along the lines of obgyn65's famous "if Firecalc says you are good to go, you are good to go," which does seem (at least to me) overconfident.

Don't get me wrong -- I think FC is useful. But there are some methodology problems. It undersamples recent years. It doesn't account for changes in bond values due to interest rate fluctuations. The most important problem is, as mentioned, that it only models a few independent time series. If I run FC over a 30 year period, you might think that it is running 113 "experiments" (the number of 30 year periods between 1871 and 2013). But there is so much overlap between the time series (does 1960-1989 really give a different result than 1961-1990?) that the number of independent experiments is much less.

I agree that with traditional (normally distributed) MC, both temporal and inter-asset correlations can be lost. That's why I'm a fan of bootstrap statistics, since it allows you to keep those correlations. I'm too lazy to describe it in detail, but Wikipedia has a decent article on it. And the results (at least based on my own calculator) are more conservative than FC, which makes me feel better.

 

[REWahoo]

I do seem to read quite a few posts along the lines of obgyn65's famous "if Firecalc says you are good to go, you are good to go," which does seem (at least to me) overconfident.

 

[photoguy]

But all those alternate realities are based on the Evil Dr. Pfau's favorite tool: made up numbers. So you don't like using the historical record and would rather base your plans on numbers Pfau essentially pulled out of his ass? Am I the only one who thinks this is problematic?

Actually I really do like using the historical record because it's less susceptible to researcher fudging and I would always look at this first. I was just saying that in terms of validation the MC researcher should put in assumptions that match the historical record and they should get results roughly the same as Firecalc. They might then explore other scenarios.

I guess my stance would be they both (historical and MC) can provide useful information but they also have problems. I'm very skeptical of some of Pfau's papers (e.g. the rising equity glide path one).

For example, there aren't "a s***load of parameters" used, there are two (mean and variance) for normal or log-normal distributions and maybe one or two others if a "fat-tailed" distribution is used. There are no parameters used at all if bootstrap sampling is used.

You say that MC uses only two parameters? But on how many variables? FC is looking at market returns, fixed returns, and inflation. I don't believe those are totally independent, so how does MC handle any correlations (positive or negative)? Without a full understanding of that, I don't know what to think about any MC run.

Take a look at this paper by Pfau "Asset Valuations and Safe Portfolio Withdrawal Rates"

Asset Valuations and Safe Portfolio Withdrawal Rates by David Blanchett, Michael S. Finke, Wade D. Pfau :: SSRN

To setup his MC simulation he has 5 different regression models for bond returns, stock returns, and inflation. I count 14 coefficients + mean/variance for 5 error terms (an additional 10 parameters) and this does not including variations on model structure or the hard upper and lower thresholds he puts in.

But, more importantly, what are the alternatives? As a famous statistician once said: "all models are wrong, but some are useful."

I'm very fond of George Box's quote. I never meant to imply that MC methods were useless but they have their drawbacks. Same as Firecalc (except the drawbacks are different).

I agree that with traditional (normally distributed) MC, both temporal and inter-asset correlations can be lost. That's why I'm a fan of bootstrap statistics, since it allows you to keep those correlations. I'm too lazy to describe it in detail, but Wikipedia has a decent article on it. And the results (at least based on my own calculator) are more conservative than FC, which makes me feel better.

I'm not sure how bootstrap would help here since you're still only resampling from a population of about 100pts. (i'm a huge of fan of resampling methods though).

 

[ERD50]

FWIW, I do think many posters here are overconfident about Firecalc. I'm not sure why you interpreted that as putting words in other's posts -- it's just my personal opinion and I expressed it as such. ...

OK, you did say 'some', not 'most'. I might be a little sensitive and have a little pent-up stuff since someone recently ran with something I said and took it in a very demeaning direction, and that thread was closed before I had a chance to respond. Sorry.

I do seem to read quite a few posts along the lines of obgyn65's famous "if Firecalc says you are good to go, you are good to go," which does seem (at least to me) overconfident.

REWahoo's emoticon is worth a thousand words. I'll just say that poster is so far from the mainstream here that anything they say is ... well, I better just stop there to keep myself out of trouble.

Don't get me wrong -- I think FC is useful. But there are some methodology problems. ...

It's a tool. It provides some useful information. It can't do what isn't in its bag of tricks (hammers make bad saws). I mentioned the bond valuation issue, though I don't yet have a handle on how much that might throw off results (I need to research that). I suspect it isn't that big of a problem, only fairly small amounts get re-balanced, it probably washes over time to some degree, and the results with 100% EQ (which would not be affected) don't seem out of line with expectations. But it is an issue.

I'm not so sure the other issues are a big deal. Yes, there is only so much history, and much of it within the same cycle. But what can you do? Does randomizing those scenarios to create more of them really add much to the party? How do we know that the MC runs would look like anything that could have happened in history.

I'll need to read up on bootstrapping, but if I'm getting a hint of it from photoguy's post, I doubt I'll be impressed.

Bottom line, I just don't see how anyone can know that the MC runs represent anything useful. Some explanations seems circular to me - they compare them to history. Then what's the point, we have history?

But more importantly - is your MC giving you a more generous WR, or a more frugal one? If you are going to be more frugal, is it any different than using history and adding some cushion? If you are going to spend more than history says, doesn't that mean the MC didn't account for what we have already seen?

Maybe I'm missing something, but MC just strikes me as trying to tell me something that it just can't tell me (just like FC can't run a 50 year sequence starting 40 years ago).

-ERD50

 

[FIRE'd@51]

If you look at Figure 1 in Pfau's paper you can see that, except for equity allocations below about 25%, the MC success rates and the Rolling History success rates are not all that different. Furthermore, I'm not sure I believe the 100% success rate for the Rolling History with equity allocations between 40% and 70%. I have never found a Firecalc result of 100% success using a 4% withdrawal rate, so perhaps the blue line should be lower, making the results even closer.

 

[ggbutcher]

In spite of my missive below, note that I regard statistics as the Evil Math...

That said, MC analysis can be useful to gain insight into sensitivity of parameters. Most MC "draws" I've worked with have actually been based on historical performance of a parameter, using its descriptive statistics, e.g., mean, standard deviation. So, the historical performance of a stock, for example, can be used to develop a statistical draw function that returns values randomly selected from the bell curve of its historical data.

The historical record of the market is but one of what were a really large number of possible outcomes. But, other things could have happened, and the outcome would be very different. Well-constructed MC analysis helps you to effectively consider that possible variability in 'guessing' future performance.

With regard to FireCalc and I-ORP, I use both the historical and MC modes, and each tell me something useful. I spend more time regarding the historical mode reports, however, because I haven't picked at either's MC methodology to know what parameters are being MC-ed...

 

[Fred123]

I'm not sure how bootstrap would help here since you're still only resampling from a population of about 100pts. (i'm a huge of fan of resampling methods though).

Bootstrapping is sampling with replacement. So, in the example I gave (30 year cycles, from 1871-2013), if you're comparing to FC, you get a "sample space" of size 113**30 or about 10**61 vs 113 for FC. That's quite a few more experiments...

 

[mathjak107]

historical data is to linked to only the results that happened before it. a great market runnup ala 1987 to 2003 could effect many many rolling time frame outcomes yet that run up of 14% for 17 years as an average may never happen again.

i like the idea monte carlo sims looked at the 10,000 possibilities of outcomes as well. the mere fact we have no historical time frame with low rates and high valuations at the same time makes any future historical sims unreliable at best if not dead wrong since no time frame existed to reflect where we are now.

in fact look at those poor results pfau got when accounting for the low yields in the monte carlo sims. even 30% stock looks poor in that run up. to be honest i believe that model may be the more useful tool right now for myself since i am retiring in 11 months 302 days 11 hours 12 minutes

 

[FIRE'd@51]

Bootstrapping is sampling with replacement. So, in the example I gave (30 year cycles, from 1871-2013), if you're comparing to FC, you get a "sample space" of size 113**30 or about 10**61 vs 113 for FC. That's quite a few more experiments...

This approach makes a lot of sense to me. It seems to to capture the best of both approaches. There is no need to estimate a covariance matrix with all its associated problems, mainly the non-stationarity of the parameters. It retains the sequential correlations from market/inflation cycles built into Firecalc while reducing the serial correlation from data overlap. If I understand bootstrapping correctly, it makes all 30-year paths equally probable in a single drawing.

Can you tell us the difference in succes rates you get using Firecalc and the MC/bootstrap process for the same portfolio mix, e.g. a 75/25 equity/bond mix?

 

[Fred123]

Can you tell us the difference in succes rates you get using Firecalc and the MC/bootstrap process for the same portfolio mix, e.g. a 75/25 equity/bond mix?

I'll take a look in a few days -- heading off soon for a a beach backpack trip in Olympic Natl. Park. Retirement activities have to come first!

 

[haha]

I'll take a look in a few days -- heading off soon for a a beach backpack trip in Olympic Natl. Park. Retirement activities have to come first!

Give my regards to Shi-Shi.

Ha

 

[photoguy]

Bootstrapping is sampling with replacement. So, in the example I gave (30 year cycles, from 1871-2013), if you're comparing to FC, you get a "sample space" of size 113**30 or about 10**61 vs 113 for FC. That's quite a few more experiments...

In bootstrapping you're just recombining the existing data to come up with alternate but similar histories.

One problem with bootstrapping is that if a phenomenon is not represented in the source data (or is inadequately represented) then all of the bootstrap samples will also be missing this behavior. So bootstrapping won't address Mathjak's concern of venturing into uncharted waters.

 

[Fred123]

Give my regards to Shi-Shi.

Ha

I will and good guess!

 

[NgineER]

lllcpb

 

[Fred123]

This approach makes a lot of sense to me. It seems to to capture the best of both approaches. There is no need to estimate a covariance matrix with all its associated problems, mainly the non-stationarity of the parameters. It retains the sequential correlations from market/inflation cycles built into Firecalc while reducing the serial correlation from data overlap. If I understand bootstrapping correctly, it makes all 30-year paths equally probable in a single drawing.

Can you tell us the difference in succes rates you get using Firecalc and the MC/bootstrap process for the same portfolio mix, e.g. a 75/25 equity/bond mix?

For those that are still interested:

I ran a comparison of FC and bootstrap for a 50/50 SP500/10 year treasury bond mix over 30 years. My FC simulator results are a little different than the online version due to different bond calculation assumptions, but they are in the same ballpark. Disclaimer: although I've done my best to make sure my simulator works correctly, there could still be some things that I have missed.

Here's the table of fail rates for various withdrawal rates using the two different simulators:

3%     3.5%     4.0%     4.5%     5%    
-----------------------------------------
0.0%   0.0%     0.0%     26%     40%        FC
1.8%   5.5%    13.5%     24%     38%        Bootstrap

The withdrawal rate for 5% failure is 4.1% for FC and 3.4% for bootstrap.

 

[clifp]

This topic has come up before, and I'm always surprised at the aversion some people here have to MC analysis and the overconfidence they have in FC (which relies on perhaps 5 independent samples if you are looking at 30 year outcomes). Some of this seems to be based on a misunderstanding on how MC works if historical data is used to model the distribution. For example, there aren't "a s***load of parameters" used, there are two (mean and variance) for normal or log-normal distributions and maybe one or two others if a "fat-tailed" distribution is used. There are no parameters used at all if bootstrap sampling is used.

But, more importantly, what are the alternatives? As a famous statistician once said: "all models are wrong, but some are useful."

There may only mean and variance. But figuring out what the mean and variance is requires plugging in hundreds of numbers and making some SWAG (scientific wild ass guess). So for instance are the average return for US equity based on starting with 1871, 1921, or 1946 (arguments can be for all dates). Or do you decide like Dr Pfau that US equity outperformed most markets (although not places like Australia and Canada) so going forward US equities should perform more like the rest of the world. Or do you decide that the stock markets in the rest of the world have evolved to look at lot more like the US so the ROW equity return should look more like the historical returns of the US.

There was the potential for a complete collapse of the economic system in the fall of 2008. So valuing stocks was a difficult challenge..By March of 2009 the danger of complete collapse had passed and we were just dealing with a deep recession. I confidently predicted at the time the market would double in five years due to fundamentals like earnings, book values, dividends, yada yada. I'm certainly not making that prediction for the next 5 years, nor are any other value investors. Likewise Warren Buffett, in summer of 1999 warned the future stock market returns for the next 5 years were going to below average.

Why because future stock market returns are highly influenced by the returns of the past couple of years. Something that Monte Carlo simulations ignore.

 

[mathjak107]

the ignoring of that influence of previous years is just what monte carlo simulations want to rule out. the previous combo's in that order will likely never play out that way again.

in fact we are in one of them now that never happened before and is uncharted , low rates and high stock valuations only appeared in monte carlo simulations prior never in historical data.