SWR's and Success Probabilities

[FIRE'd@51]

We all know that the rolling 30-year periods used in FireCalc are not statistically independent due to data overlap.

I'm not a statistician, but I've become increasingly concerned that the SWR success rates from FireCalc may be significantly overestimated, due to this overlap which introduces significant serial correlation into the data series. I do know that serial correlation introduces a downward bias in a standard deviation calculation, and it's standard deviation that leads to the success probabilities. This may partially explain why Monte Carlo simulations, which use a standard deviation calculated from non-overlapping data intervals (e.g. annual returns) tend to give lower SWR's than FireCalc. IOW, I'm suggesting that a 4% SWR could actually lead to a significantly lower success rate than 95% due to this data overlap effect; and, since we really don't know what the probability of success is, one should rely upon the FireCalc SWR that has never failed.

[donheff]

I guess I don't understand why the math about correlation would would matter. Firecalc is used to discover how you would have faired in any X period in financially recorded history. The answer is what it is. Whether you assume that the overlap you are concerned with is likely a feature of the world and thus likely to be repeated in the future is another matter.

[Midpack]

FIRECALC doesn't "overestimate" anything, it's based on actual history. I wouldn't consider Monte Carlo superior by any means, it's another worthwhile method to consider that's all, but YMMV.

Quote:

Originally Posted by FIRECALC How can FIRECalc predict future returns from past performance? It can't. And it doesn't try. In fact, it tries to predict what will not happen. This might sound confusing, but it's really simple. Consider an analogy: Suppose you are building a house in Honolulu. How do you decide how much heating and air conditioning capacity you will need? We know that the lowest it has ever been there was 52°, on a day in February 1902 and again on another day in January 1969, and the hottest it has ever been was 95°, in 1994. Buying a system suitable for an Anchorage-style winter and a Phoenix-style summer would be a major waste of money that could be better used elsewhere. FIRECalc works the same way, using stock market history and your portfolio and spending plan instead of weather history and furnace capacity. No one could predict the temperature for any specific given future date during the decades that house will be used, and no one can predict the future returns of your investments. But by knowing the historical worst cases, you'll have the information to judge if your savings are sufficient to handle the winter.

[ERD50]

Quote:

Originally Posted by donheff I guess I don't understand why the math about correlation would would matter. Firecalc is used to discover how you would have faired in any X period in financially recorded history. The answer is what it is. Whether you assume that the overlap you are concerned with is likely a feature of the world and thus likely to be repeated in the future is another matter.

Agree with previous posts. It is what it is. It reports what would have happened if you retired in 1966, or in 1965, etc. So 29 of those years overlapped? They did in real life as well.

And the way most of use FIRECalc (I assume), is to focus on the failures. So most of the data isn't even taken into consideration. Sure, it's nice to know that some X% of the time we would have ended up with a high multiple of our initial buying power, or that a significant % of the time we would have maintained our buying power, but the failures are the main concern. And they are what they are.

-ERD50

[photoguy]

As others have mentioned FIRECALC reports on the exact number of runs in HISTORY that would have failed.

It doesn't produce what the probability of success is going forward (which is THE NUMBER that everyone wants). If you want to use the FIRECALC number as a forward estimate you should probably be concerned with two types of errors:

Bias which would be if the historical Firecalc success rate systematically under or over-estimates the probability going forward. I think many people believe that Firecalc is biased to produce higher success rates (as a future estimate) because currently PE10 is high right now and bond yields are low. This is why so many people take a haircut on the allowable withdrawal rate (e.g., instead of 4% I'll use 3.5%).

Variance which would be due to fluctuations in the observed rate due to chance (but not systematic errors). For example, if you flipped a coin 10 times you might get 4 heads and estimate the probability of getting heads as 40%. In this case, due to variance the estimate is 10% too low.

Firecalc does not produce any sort of confidence bounds. To deal with this, you could mentally add +/- 10% to whatever number firecalc issues (I guess someone could calculate the impact of the overlapping periods by running simulations to figure out a better estimate than this).

[Lsbcal]

Quote:

Originally Posted by FIRE'd@51 ...This may partially explain why Monte Carlo simulations, which use a standard deviation calculated from non-overlapping data intervals (e.g. annual returns) tend to give lower SWR's than FireCalc. IOW, I'm suggesting that a 4% SWR could actually lead to a significantly lower success rate than 95% due to this data overlap effect; and, since we really don't know what the probability of success is, one should rely upon the FireCalc SWR that has never failed.

I don't know what is built into Monte Carlo simulators. Clearly MC simulation is not doing real world events and worst cases may pop up then could ever happen in the real economic world. Do they allow many overlapping series? I would imagine the design and the user choices is key to how good the simulator is.

It's true the FIRECalc has many sequences with common years like the 1929-1932 period. I find it best to just go for 100% success rate with a reasonable floor on the minimum portfolio value during the entire simulation. If FC fails, I want to know why and what years were responsible. That is where the spreadsheet output can be very handy.

I was plunking around and found this simple MC simulator from Vanguard. Notice there is no portfolio minimum value besides zero allowed: https://retirementplans.vanguard.com...estEggCalc.jsf

[REWahoo]

Quote:

Originally Posted by photoguy As others have mentioned FIRECALC reports on the exact number of runs in HISTORY that would have failed. It doesn't produce what the probability of success is going forward (which is THE NUMBER that everyone wants). If you want to use the FIRECALC number as a forward estimate you should probably be concerned with two types of errors: Bias which would be if the historical Firecalc success rate systematically under or over-estimates the probability going forward. I think many people believe that Firecalc is biased to produce higher success rates (as a future estimate) because currently PE10 is high right now and bond yields are low. This is why so many people take a haircut on the allowable withdrawal rate (e.g., instead of 4% I'll use 3.5%). Variance which would be due to fluctuations in the observed rate due to chance (but not systematic errors). For example, if you flipped a coin 10 times you might get 4 heads and estimate the probability of getting heads as 40%. In this case, due to variance the estimate is 10% too low. Firecalc does not produce any sort of confidence bounds. To deal with this, you could mentally add +/- 10% to whatever number firecalc issues (I guess someone could calculate the impact of the overlapping periods by running simulations to figure out a better estimate than this).

You forgot the third type of "error":

Asteroid Strike which would likely result in portfolio failure regardless of whatever fudge factor you added to FIRECalc.

[Chuckanut]

Quote:

Originally Posted by REWahoo You forgot the third type of "error": Asteroid Strike which would likely result in portfolio failure regardless of whatever fudge factor you added to FIRECalc.

Ah, yes. The exogenous events that throw the best plans under the bus.

The only way to deal with these events and their effect on FireCalc is to "Trust but Verify". And pivot if necessary.

[Midpack]

Quote:

Originally Posted by REWahoo You forgot the third type of "error": Asteroid Strike which would likely result in portfolio failure regardless of whatever fudge factor you added to FIRECalc.

I always thought this (below) might be you...so you've moved from comet to asteroid?

Quote:

Originally Posted by FIRECALC How can I be sure these results will work in the future? FIRECalc's standard model uses the overall US stock market performance. Most 401k and similar retirement plans offer investment choices ("index funds") that are closely tied to the overall market performance, and the others generally tell you how they compare. If the next few decades are even worse for the stock market than the worst that has ever been seen, including the Great Depression, then all bets are off. But as one early retiree pointed out, there isn't much anyone can do to prepare for a comet hitting us!

[ERD50]

Quote:

Originally Posted by photoguy ... Bias which would be if the historical Firecalc success rate systematically under or over-estimates the probability going forward. I think many people believe that Firecalc is biased to produce higher success rates (as a future estimate) because currently PE10 is high right now and bond yields are low. This is why so many people take a haircut on the allowable withdrawal rate (e.g., instead of 4% I'll use 3.5%). ...

But...

That 4% succeeds in all but the worst 5% of the periods in the database. So I think the relevant question is whether the current market is more bloated then it was in those few periods.

We aren't really looking at averages, so I don't think it is relevant whether the market is more overheated than average, just if it is hotter than the 'worst' times of the past.

For reference, I use a 100% success rate and 45 years for my target. One of us may live that long, and I don't want to use a plan that has been known to fail in the past. Still, the future could be worse than the worst of the past. Hopefully, I recognize that, and adjust in time.

-ERD50

[Bestwifeever]

Quote:

Originally Posted by REWahoo You forgot the third type of "error": Asteroid Strike which would likely result in portfolio failure regardless of whatever fudge factor you added to FIRECalc.

You just have to keep your portfolio covered with a good grade of tin foil.

[Huston55]

Quote:

Originally Posted by photoguy As others have mentioned FIRECALC reports on the exact number of runs in HISTORY that would have failed. It doesn't produce what the probability of success is going forward (which is THE NUMBER that everyone wants). If you want to use the FIRECALC number as a forward estimate you should probably be concerned with two types of errors: Bias which would be if the historical Firecalc success rate systematically under or over-estimates the probability going forward. I think many people believe that Firecalc is biased to produce higher success rates (as a future estimate) because currently PE10 is high right now and bond yields are low. This is why so many people take a haircut on the allowable withdrawal rate (e.g., instead of 4% I'll use 3.5%). Variance which would be due to fluctuations in the observed rate due to chance (but not systematic errors). For example, if you flipped a coin 10 times you might get 4 heads and estimate the probability of getting heads as 40%. In this case, due to variance the estimate is 10% too low. Firecalc does not produce any sort of confidence bounds. To deal with this, you could mentally add +/- 10% to whatever number firecalc issues (I guess someone could calculate the impact of the overlapping periods by running simulations to figure out a better estimate than this).

It's true that current PE10 is high historically, in the range if the historical highs. But, that's why we use >=95% success rates and analyze any failures. To call the current situation 'biased' seems to be counter to the methodology and overly pessimistic.

S&P 500 PE Ratio

Quote:

Originally Posted by REWahoo You forgot the third type of "error": Asteroid Strike which would likely result in portfolio failure regardless of whatever fudge factor you added to FIRECalc.

Revising The Math: Chances Of Asteroid Hitting Earth Greater Than Once Thought « CBS Baltimore

Evidently, not as remote a possibility as we previously thought. Damn it REW, now I have to change my spreadsheet.

"Figuring in those smaller asteroids, the air-burst like the one in Russia has dropped from a possibility of once every 150 years to once in every 30 years."

However, if Congress was in session during such an event, this might cause a dramatic rise in portfolio values.

"It prompted NASA to do a quiet tabletop drill, where such a hit on D.C. was predicted to kill 78,000 people. An asteroid smaller than a football field might do it."

[W2R]

I don't think that any of the methods to help us determine withdrawal rate are reliable enough to bet my life and future on. What these methods (including FIRECalc) provide to me, is a starting point. Should I withdraw 20%, or 0.00001%? FIRECalc gives me a starting point in the decision making process and what I do with it depends on me.

FIRECalc is only telling us what happened in the past. It really says nothing about the future IMO. The assumption that the future would mirror the past in a probabalistic sense, is ours to make or to not make.

Perfect, no. Better than many other methods (including my crystal ball), yes.

[Ed_The_Gypsy]

Quote:

Originally Posted by Lsbcal I don't know what is built into Monte Carlo simulators. Clearly MC simulation is not doing real world events and worst cases may pop up then could ever happen in the real economic world. Do they allow many overlapping series? I would imagine the design and the user choices is key to how good the simulator is. It's true the FIRECalc has many sequences with common years like the 1929-1932 period. I find it best to just go for 100% success rate with a reasonable floor on the minimum portfolio value during the entire simulation. If FC fails, I want to know why and what years were responsible. That is where the spreadsheet output can be very handy. I was plunking around and found this simple MC simulator from Vanguard. Notice there is no portfolio minimum value besides zero allowed: https://retirementplans.vanguard.com...estEggCalc.jsf

Cool! I had not seen this on Vanguard's web site. Thanks!

Visually, this shows how important the first years are.

Also, the importance of re-evaluating after 4 or 5 years. Shortening the duration from 30 years to 25 years does not change the survivability very much. One 75/25/0 4.5% scenario showed 84% survivability in 30 years and 89% in 25 years.

Many nice tools these days.

[FIRE'd@51]

Quote:

Originally Posted by photoguy As others have mentioned FIRECALC reports on the exact number of runs in HISTORY that would have failed. It doesn't produce what the probability of success is going forward (which is THE NUMBER that everyone wants). If you want to use the FIRECALC number as a forward estimate you should probably be concerned with two types of errors: ... Variance which would be due to fluctuations in the observed rate due to chance (but not systematic errors). For example, if you flipped a coin 10 times you might get 4 heads and estimate the probability of getting heads as 40%. In this case, due to variance the estimate is 10% too low. Firecalc does not produce any sort of confidence bounds. To deal with this, you could mentally add +/- 10% to whatever number firecalc issues (I guess someone could calculate the impact of the overlapping periods by running simulations to figure out a better estimate than this).

This is what I was getting at. Many financial planners (and others) use the 4% WR and 95% success probability as a predictor of a future probability even though they are based upon only about five independent 30-year data sets, similar to your coin-flip example. I haven't seen any rigorous research along the lines you suggest.

[Midpack]

Quote:

Originally Posted by FIRE'd@51 This is what I was getting at. Many financial planners (and others) use the 4% WR and 95% success probability as a predictor of a future probability even though they are based upon only about five independent 30-year data sets, similar to your coin-flip example. I haven't seen any rigorous research along the lines you suggest.

I think we agree with you. But there have been several "rigorous" studies about future withdrawal rates with the upshot typically "3% is the new 4%" - coincidentally there's another thread on the topic just started here. And I am sure there will be more research. Most factor in prolonged low bond yields and interest rates and the current above average equity market valuations to arrive at more conservative WR estimates. Even using 100% success rates in FIRECALC isn't an answer as your OP suggests.

But again FIRECALC does not predict the future, and that's very clearly stated. So FIRECALC doesn't "significantly overestimate" or give incorrect success probabilities. It simply plug-n-chugs history and spits out those results for consideration.

To fault an elegant "app" that the authors offer at no cost (few donate anything at all) for providing answers it doesn't pretend to offer may be undeserved. I tried to write my own FIRECALC, if you think it's easy, give it a whirl. But for anyone who still may erroneously think FIRECALC predicts anything, hopefully your thread will help them better understand.

Don't forget while a 95% success rate means you have a 1 in 20 chance of failing if you withdraw X%, it also means you have a 19 in 20 chance of withdrawing more than X%. And withdrawal rates are an axe, not a scalpel. You will undoubtedly have to make adjustments along the way...

[FIRE'd@51]

Quote:

Originally Posted by Midpack To fault an elegant "app" that the authors offer at no cost (few donate anything at all) for providing answers it doesn't pretend to offer may be undeserved

I am not faulting FireCalc. To the contrary, I think it's the best program around. I am only wondering if there is a way to manipulate the overlapping historical data to better simulate a statistical ensemble of "independent" data points. This has nothing to do with PE10 or other valuations, which will always be present. I believe these affect the mean of the ex-ante distribution, but not the variance. I'm interested in focusing soley on the statistical analysis which, I believe, can give us a better handle on the variance, which ultimately determines the ex-ante probabilities.

[Midpack]

Quote:

Originally Posted by FIRE'd@51 I am not faulting FireCalc. To the contrary, I think it's the best program around. I am only wondering if there is a way to manipulate the overlapping historical data to better simulate a statistical ensemble of "independent" data points. This has nothing to do with PE10 or other valuations, which will always be present. I believe these affect the mean of the ex-ante distribution, but not the variance. I'm interested in focusing soley on the statistical analysis which, I believe, can give us a better handle on the variance, which ultimately determines the ex-ante probabilities.

Fair enough, let us know if you find a more meaningful method for analyzing historical returns and withdrawals rates. I am sure many here would be interested...

[FIRE'd@51]

Quote:

Originally Posted by Midpack Fair enough, let us know if you find a more meaningful method for analyzing historical returns and withdrawals rates. I am sure many here would be interested...

Midpack

I find your tone a bit offensive. I started this thread with the hope that one (or more) of the thousands of readers of this forum better versed in statistics than myself would suggest a way to do this, or even if it can be done. If I knew how to do it myself, I would have done it already.

[Midpack]

Quote:

Originally Posted by FIRE'd@51 Midpack I find your tone a bit offensive. I started this thread with the hope that one (or more) of the thousands of readers of this forum better versed in statistics than myself would suggest a way to do this, or even if it can be done. If I knew how to do it myself, I would have done it already.

Sorry about that, but you're reading in "tone" that's not intended. I tend to err on the side of being direct, sometimes that's helpful but I realize sometimes it's decidedly not helpful. We/I would be sincerely interested on better/other studies, but for whatever reason I felt your OP read as potentially undue criticism of FIRECALC as did other members. Evidently we both misread each other a bit...