SWR's and Success Probabilities

[donheff]

Quote:

Originally Posted by FIRE'd@51 Midpack I find your tone a bit offensive. I started this thread with the hope that one (or more) of the thousands of readers of this forum better versed in statistics than myself would suggest a way to do this, or even if it can be done. If I knew how to do it myself, I would have done it already.

I still don't understand how you could do something to statistically improve an algorithm that specifically eschews statistics for arriving at its results. If you statistically removed the correlations, outliers, or any other historical facts you would no longer have a historical calculator.

[MichaelB]

Quote:

Originally Posted by FIRE'd@51 I am not faulting FireCalc. To the contrary, I think it's the best program around. I am only wondering if there is a way to manipulate the overlapping historical data to better simulate a statistical ensemble of "independent" data points. This has nothing to do with PE10 or other valuations, which will always be present. I believe these affect the mean of the ex-ante distribution, but not the variance. I'm interested in focusing soley on the statistical analysis which, I believe, can give us a better handle on the variance, which ultimately determines the ex-ante probabilities.

Not a statistician, but what other options are there in addition to looking at either historical cycles or random rates of return? Wouldn't variance from either of these just be adding bias and reducing the value of the output?

[ERD50]

Quote:

Originally Posted by FIRE'd@51 I ... I am only wondering if there is a way to manipulate the overlapping historical data to better simulate a statistical ensemble of "independent" data points. This has nothing to do with PE10 or other valuations, which will always be present. I believe these affect the mean of the ex-ante distribution, but not the variance. I'm interested in focusing soley on the statistical analysis which, I believe, can give us a better handle on the variance, which ultimately determines the ex-ante probabilities.

I'm lost as to how this could be done, or how there could be any validity to the results. I agree that the data points really only represent a handful of patterns. And I think that's about all we have.


edit/ - cross posted with the others

-ERD50

[FIRE'd@51]

Quote:

Originally Posted by donheff I still don't understand how you could do something to statistically improve an algorithm that specifically eschews statistics for arriving at its results. If you statistically removed the correlations, outliers, or any other historical facts you would no longer have a historical calculator.

I didn't say remove correlations - I said remove the serial correlation from the overlap. They are totally different things. For example, if the year were 5013 instead of 2013, we would have approximately 105 independent non-overlapping 30-year data points, which would be a better sample size to try to estimate future probabilities from, analogous to flipping a coin 105 times instead of 5 times to estimate the probability of heads.

[FIRE'd@51]

Quote:

Originally Posted by ERD50 I'm lost as to how this could be done, or how there could be any validity to the results. I agree that the data points really only represent a handful of patterns. And I think that's about all we have.

Maybe it can't be done, although I have to believe there are statistical methods that attempt to deal with the overlap problem, as it must occur often in time-series data. As I said above to Midpack, I started this thread with the hope that someone better versed in statistics than myself could opine on this subject.

[samclem]

Quote:

Originally Posted by FIRE'd@51 IOW, I'm suggesting that a 4% SWR could actually lead to a significantly lower success rate than 95% due to this data overlap effect; and, since we really don't know what the probability of success is, one should rely upon the FireCalc SWR that has never failed.

Even it were possible to add a greater amount of "cases" to the retrospective analysis (e.g. by disaggregating the present US data to make the years independent and nonsequential, adding data from Australia or Canada, etc), I think it would introduce more uncertainties than it resolves. More fundamentally, it would remain retrospective analysis with all the inherent uncertainty about its applicability to the future.
We say that "past results are not indicative of future returns", but in fact that's something each of us is forced to ignore, to a greater or lesser extent, as we plan for our future. Given that we are breaking the rules in this way, how much will be gained, in a practical sense, by improving the accuracy of the FIRECalc results? They provide a rough starting point.

[Lsbcal]

Quote:

Originally Posted by FIRE'd@51 I didn't say remove correlations - I said remove the serial correlation from the overlap. They are totally different things. For example, if the year were 5013 instead of 2013, we would have approximately 105 independent non-overlapping 30-year data points, which would be a better sample size to try to estimate future probabilities from, analogous to flipping a coin 105 times instead of 5 times to estimate the probability of heads.

Maybe you have touched on the problem yourself. How many years are needed to separate world economic events that affect US stock and bond returns? And so how many years to find independent periods to group together into a neat statistical study where we can apply STAT 101?

My guess is you will need a lot of years to reduce correlations towards zero -- at least several decades. If we really want to be conservative, maybe 100 years. For instance, WW1 is thought to have had a lot to do with the 1930's Great Depression. So maybe the events and economics of 1914 related to the economics 20 years later. My feeling is that the 1930's economics (80+ years ago) are still somewhat a ghostly presence in our current thought processes. I certainly consider the 1930's when deciding on our spending strategies.

[haha]

Quote:

Originally Posted by FIRE'd@51 Midpack I find your tone a bit offensive. I started this thread with the hope that one (or more) of the thousands of readers of this forum better versed in statistics than myself would suggest a way to do this, or even if it can be done. If I knew how to do it myself, I would have done it already.

It can't really be done. Either something is there, in the data, or it is not. In this case, it is not. The Monte Carlo comes at the problem from an entirely different direction, and of course has its own set of limitations.

Ha

[photoguy]

Quote:

Originally Posted by Huston55 It's true that current PE10 is high historically, in the range if the historical highs. But, that's why we use >=95% success rates and analyze any failures. To call the current situation 'biased' seems to be counter to the methodology and overly pessimistic.

I'm not saying the current economic situation is biased. What I am trying to say is that if one uses the number straight out of firecalc (say 5% failure) and believes that starting today their chance of failure is also 5% they are likely underestimating the true probability.

Quote:

Originally Posted by FIRE'd@51 I am only wondering if there is a way to manipulate the overlapping historical data to better simulate a statistical ensemble of "independent" data points. .

I think this is what the creators of MC simulations are trying to achieve. They realize the limitations of the historical data record and setup a simulation with the parameters of their model informed by the past data. If they've done a good job, then hopefully they've captured all the important behaviors in the historical data (I have my doubts about how well this can be done).


Another thing one could do is take a MC simulation (which doesn't need overlapping data points for each 30 year run) and do so many runs that you get a very precise measure of the failure rate. Then modify the simulation to be like firecalc (overlapping 30 years runs, ~100 cycles) and get an idea of the distribution of results. This wouldn't let you improve the results but would at least give you a better idea of the range of error due to the overlapping runs.

[donheff]

Quote:

Originally Posted by FIRE'd@51 I didn't say remove correlations - I said remove the serial correlation from the overlap. They are totally different things. For example, if the year were 5013 instead of 2013, we would have approximately 105 independent non-overlapping 30-year data points, which would be a better sample size to try to estimate future probabilities from, analogous to flipping a coin 105 times instead of 5 times to estimate the probability of heads.

OK, I think I get what you want but it would be a different sort of calculator from Firecalc. Instead of evaluating how your portfolio would have done against all actual x year historical periods you would want to simulate how future 30 year periods might statistically play out given the historical record we have to date. But isn't that precisely how some of the Monte Carlo simulators are constructed? Why not just look at how they describe their approaches and pick one that does what you want.

[Animorph]

There is not enough data to do any better historically.

Ed Easterling at Crestmont Research was the first person I know of to look at starting conditions for retirement. He used only P/E though. He actually joined this forum for a while, though he did not receive a lot of support from most here.

Crestmont Research: Financial Market and Economic Research

At one time he had a nice spreadsheet of returns versus starting P/E, but I don't see it in a quick browse of his website.

[Lsbcal]

Quote:

Originally Posted by Animorph ... At one time he had a nice spreadsheet of returns versus starting P/E, but I don't see it in a quick browse of his website.

The data is fairly easy to get. Just take the Shiller spreadsheet and calculate the trailing P/E. Then take the FIRECalc run spreadsheet data.

But how to present this is perhaps the harder part. P/E is one point and from that retirement year we get a curve from FIRECalc going out maybe 30 years. One would want the low point on any curve and maybe also the final portfolio value. Guess that is a nice task for someone here.

[haha]

Quote:

Originally Posted by Animorph There is not enough data to do any better historically. Ed Easterling at Crestmont Research was the first person I know of to look at starting conditions for retirement. He used only P/E though. He actually joined this forum for a while, though he did not receive a lot of support from most here.

He was insufficiently respectful of our forum gods.

I have read two of his books, he seems very good to me. Though when markets are boom bust as they have been, and IMO must continue to be as long as we have this massive FRB intervention that we have had since 2009, his work can't catch it. IMO nothing other than momentum can deal with this, and of course this has its own difficulties.

Ha

[akck]

I think there is too much wishing to improve accuracy when there is no way to do so in a meaningful way. As was said, past performance is not a predictor of future results. We have no way to know what the distant future holds. There are just too many unknown future world events that will affect your results.

In addition, we aren't static. I don't believe anyone will go blindly forward taking their SWR as events happen. Adjustments up and down will be made to reflect changes to your situation.

Firecalc is just a tool to help plan your retirement. Like horseshoes and hand grenades, it just needs to be close, that you're in the ballpark of having enough.

[CaliforniaMan]

Very interesting thread. My problem is that I seem to agree with everyone on this. FIRE'd@51 has a very good point, we are looking at a lot of overlapping data, and that will underestimate the variation.

On the other hand Monte Carlo approaches can for example have you going from 2% inflation one year to 18% inflation the next, back to 2% the next year, same with interest rates, etc. We can model the deltas from year to year, which would tend to be more realistic, but then you could get some years with really really high or low values. Unrealistic high variation.

As many have pointed out here, and I think I agree, the overlapping time period approach as use by FireCalc and others is probably the best we can use. Just to get a baseline of what may be "reasonable" and then adjust as necessary.

The real problem in all of this is of course, that we do not have a stationary time series. Is 1900s really the same as 2000s? Will the rate of technology change be the same? Will the US still be as important as we were for most of this period? Will it matter?

And even if we had data back many more decades or centuries would it make any difference to the validity of our projections?

And then what about that asteroid strike? I hear that all of the gold we have found has actually come from past meteor strikes. (The original primordial gold sank to the core along with the other heavy elements) What if it contains tons and tons of gold, or maybe diamonds? Hmmm....

By the way, this is my first post as a new member.

[Fred123]

Another Firecalc problem that hasn't been mentioned is undersampling of recent years. If, for example, you are looking at 30 year periods then the 2008 meltdown is only included in 5 time series, while years 1982 and earlier are included 30 times. And, of course, it get worse the longer the time period you are using in the calculator. I'm guessing that the underweighting of 2008 makes Firecalc overoptimistic.

I wrote my own little calculator using three methods: Firecalc, MC analysis assuming a normal distribution, and MC analysis using resampling with replacement (analagous to bootstrap sampling). Firecalc is much more optimisitic than the MC analysis, so I tend to be more comfortable with MC.

[Midpack]

Quote:

Originally Posted by Fred123 Another Firecalc problem that hasn't been mentioned is undersampling of recent years. If, for example, you are looking at 30 year periods then the 2008 meltdown is only included in 5 time series, while years 1982 and earlier are included 30 times. And, of course, it get worse the longer the time period you are using in the calculator. I'm guessing that the underweighting of 2008 makes Firecalc overoptimistic.

True. However the actual Great Depression (which was far worse than 2008) is included 30 times, and that hasn't happened again for about 80 years (let's hope it never does). And the worst 30 years periods since 1871 to present began 1965-69 and 1973 (ending 2003), which includes neither the 1930's Great Depression or the 2008 Meltdown (yet, though the market already more than recovered those losses).

FIRECALC is history, it's not optimistic or pessimistic in that it makes no predictions for the future whatsoever.

Again, retirement calculators tools are axes, not scalpels - even in the most gifted academic hands. And even if you could nail down portfolio returns, there are so many other variable unknowns you'd still be left with statistical probabilities...

[samclem]

Quote:

Originally Posted by Fred123 I wrote my own little calculator using three methods: Firecalc, MC analysis assuming a normal distribution, and MC analysis using resampling with replacement (analagous to bootstrap sampling). Firecalc is much more optimisitic than the MC analysis, so I tend to be more comfortable with MC.

Would it be best to simply append data from previous years to the end of the real-world set? So, a 30 year sample could be made from 1984-2012 (29 years)plus 1983. Another set would be 1985-2012 plus 1983-1984. Yes, there would be a big discontinuity where the "splice" occurs, but it is no worse than the discontinuity in >every< year-to-year of a typical MC simulation, and the payoff is that you can in this way include the very significant 2008-2012 data in a larger number of "runs."

[RobLJ]

Quote:

Originally Posted by samclem Would it be best to simply append data from previous years to the end of the real-world set? So, a 30 year sample could be made from 1984-2012 (29 years)plus 1983. Another set would be 1985-2012 plus 1984-1984. Yes, there would be a big discontinuity where the "splice" occurs, but it is no worse than the discontinuity in >every< year-to-year of a typical MC simulation, and the payoff is that you can in this way include the very significant 2008-2012 data in a larger number of "runs."

I'm not sure how FireCalc uses it's data set, exactly. My impression is that it runs SWR against consecutive year sets as long as the length of the withdrawal period, but I'm likely wrong.
If so, one could randomize (or Monte Carlo) consecutive 5 or 7 year sets of data, which would hybridize the historical data set against the single year Monte Carlo approach. I chose 5-7 years simply because that period would hold a typical market cycle +.
I've probably got this wrong and maybe FireCalc already uses this approach.

[Options]

There's a very good (recent) thread on the Bogleheads site regarding the use (and reliability) of various calculators. One in particular lists calculators (including FC) from most pessimistic to least. See this Boglehead site for a listing of several calculators, their attributes, and methodologies ranging from MC to deterministic.

Retirement calculators and spending - Bogleheads

I also recall reading here somewhere that a poster stated he would trust no single input to determine a comfort level for FIRE, and I tend to agree. I've run several scenarios/calculators which all tell me I'm on track. Next spring, I intend to buy ESPlanner (recommended as most detailed and helpful). If it concurs with the research I've done thus far--and I believe it will--I'll pull the plug and book that trip down the Amazon...