Suggestion: Sensitivity Analysis

modlair

Recycles dryer sheets
Joined
Apr 1, 2006
Messages
190
Accept or reject as you wish, sir:

Many SWR discussions I've read talk about the results of 4-5% being dependent on history and then debate begins about whether or not the future will be more bullish or more bearish than the past.

Perhaps a sensitivity analysis in Firecalc has merit? Meaning, someone who wants to see the effect of a more bearish future (than the past) could bias all historical numbers to be more bearish on average by X%. Or more bullish by X%.

One can then start to draw conclusions about how sensitive an SWR's success % is to a future that is biased X% up or down wrt history.

Shrug.
 
rodmail said:
Perhaps a sensitivity analysis in Firecalc has merit?  Meaning, someone who wants to see the effect of a more bearish future (than the past) could bias all historical numbers to be more bearish on average by X%.  Or more bullish by X%.
Hacker's sensitivity analysis:
Start with the SWR that produces a 95% success rate and raise it until the success rate declines to 50%.
 
rodmail said:
Accept or reject as you wish, sir:

Many SWR discussions I've read talk about the results of 4-5% being dependent on history and then debate begins about whether or not the future will be more bullish or more bearish than the past.

Perhaps a sensitivity analysis in Firecalc has merit? Meaning, someone who wants to see the effect of a more bearish future (than the past) could bias all historical numbers to be more bearish on average by X%. Or more bullish by X%.

This is great idea. The same concept can be emulated as Nords says by framing your results until you reach the probability you are comfortable with but it would be tedious.

A REALLY useful feature would be to generate a 95% confidence interval around the results, e.g. probability of succes = 92% (CI95 = 85% to 97%). That is how I get the best sense of how usable the results are for me.

Really, though, it is unrealistic to expect poor Dory to do this, and most users who are not hopeless statistics-geeks like me would not value it much.

I really love the upgrades so far. Thanks, Dory.
 
Also, if you select one of the options that iterates for a given success rate, or the one that lets you see the effects of deferring retirement, the chart will show you a rudimentary sensitivity chart.
 
This is great idea. The same concept can be emulated as Nords says by framing your results until you reach the probability you are comfortable with but it would be tedious.

Well, maybe not so great.

The alternate suggestion explores the effect on success of varying SWR, which is sort of just automating what people do manually now.

I was interested in how sensitive a given SWR's success is to the future varying in performance wrt the past.

But now that I think about this, maybe we already have it? If I think the future will be 2% worse than the past, is it legit to simply set the AER to 2.18? And if more bullish then -1.82? I checked, you can enter a negative AER.

Maybe this can be handled with just some text next to the fund expense 0.18 number.
 
Okay, wait a minute. I just crunched a few runs to explore the effect of bias on returns using the Annual Expense Ratio value that defaults to 0.18%. I nudged it to 1.18 and 2.18 etc. The effect is enormous. Adding 1% to that (simulating 1% worse returns than history) reduces a given SWR success number from 95% to 82%. Another 1% bias on returns takes it down to 70%. This is a very sensitive parameter.

But.

I think this is bogus. That number is also presumably changing the returns on the fixed income portion of the 55/45 mix. I think what we want is something that just defines S&P bias. In fact, one could make a strong case for every 1% bias on the S&P should adjust the rate bias on fixed income inversely.
 
Yes, the expense ratio is for the whole portfolio. And yes, it is a very sensitive number, as when you move 1-2% up or down, and the returns are often in the 5% range, then you're swinging performance by a huge amount.

But I'm not sure I follow re moving fixed income expenses inversely with equity expenses.

Or are you talking solely about exploring a different question, using this as a way to do so?
 
No Dory, it's still the original issue. The ability to explore an opinion that the future will be more or less bullish than the past would need to be a number applied only to the S&P.

I tried to do that with current capability by using the AER, but that's bogus for the reasons you just confirmed.

I'm only musing about inversely adjusting fixed income vs S&P because of a popular presumption that the market climbs when rates drop, and vice versa. When there is worry that the Fed is not done raising (in the current cycle), the market drops.

But I think that if you are willing to add this feature, the way to do it would be seperately for equity and bonds. If someone thinks the market will do better than historically (maybe because of the productivity available from computers in general) he can add 1% to just the equity portion of the mix. And vice versa. If you like, provide the same option for the fixed income (bonds) portion as well. Maybe someone thinks rates will be higher than in the past, or lower.

I suspect it's easy coding to do and sensitivity analysis is a good thing for this parameter.
 
rodmail said:
I suspect it's easy coding to do
Well, if it was hard then he'd do it, but since it's easy then you can pick up your coding assignment from Dory tomorrow!
 
This may need to be one of those things on the list for more careful study rather than just jumping in and doing it.

Probably showing I am not quite following the suggestion, here are some questions that immediately jump to mind when I think of making the change:

If the S&P growth each year is increased by a user-defined amount, is the dividend yield reduced to maintain the same doillar dividends, or are the dividends increased also?

Is the change additive or a factor? If additive, it obviously must be computed each year based on the magnitude of the underlying value. If a factor, wouldn't the increase in volatility create a whole new set of issues?

Several other questions are percolating, and I'd have to do more work than is allowed on Saturday afternoons to figure out if they are real or imagined, and if real, to articulate them.

Maybe if you walk through a step by step flow of the calculations..

dory36
 
dory36 said:
If the S&P growth each year is increased by a user-defined amount, is the dividend yield reduced to maintain the same doillar dividends, or are the dividends increased also?
Is the change additive or a factor? If additive, it obviously must be computed each year based on the magnitude of the underlying value. If a factor, wouldn't the increase in volatility create a whole new set of issues?
Bogle's blog has a speech from last month that breaks stock returns down into dividend return, growth return, and speculative return.

http://www.johncbogle.com/speeches/JCB_West Point 4-06.pdf

Maybe dividend & growth should be held at their historical values and the speculative return could be added (not multiplied).
 
I like your idea.

Please bring the commented code to tomorrow's meeting. Thanks!
 
Maybe if you walk through a step by step flow of the calculations..

Fair enough, and I certainly don't want to distract from other work. But your core question is EXCELLENT. It focuses on the fact that a 2% up bias in returns does NOT mean we simply increase the yearly index by 2%.

The theory is simply that in discussion of SWR, people say that 4% yielding safety really has to be 3% because returns of the future will be lower. Or they say the reverse. The question is, what impact on SWR safety does a bias upward or downward on historical equity returns have? The focal word in that paragraph is "returns".

This will NOT be simply saying . . . I think returns in the future will be 2% better than the past and therefore all that need be done is take the index value from 1871 thru 2005 and increment each year by 2%. That is wrong. That is not an increase in returns. That is an increase in the index. It is the wrong derivative in calculus.

I therefore think that 2% will have to compound over each 135 yearly value. It is returns that tell the tale for us, not the value of the index used to measure equity performance.

I was under the impression that the equity performance for Firecalc each year since 1871 (or whatever) is a published number from a historical database. That database is presumably total return (including dividends) -- meaning when one says the S&P did +5% in 2005, that includes dividends. The Dec 31 S&P index number should be 5% greater than Jan 1. The keepers of the index include dividends, I think so you would have nothing to address because their effect is in the published number.

If what you have is a closing price for an index since 1871, and not a database of yearly returns, then the return must be derived. After that we'd need:


Something like:

Return[1872] = (Index[1872] - Index[1871]) / Index[1871]
/* this computes 1872's stock market return */

BiasedReturn[1872] = Return[1872] + UserBias
/* This biases that return upward or downward */

BiasedIndex[1872] = BiasedIndex[1871] * (1.0 + BiasedReturn[1872])
/* This creates a new index. 1872's value would have been something different if the bias had been present. Note it is based on the BiasedIndex value of the previous year, because this effect will compound */


Somewhere above this loop you'd have a BiasedIndex[1871] = Index[1871] because it is the first datapoint and needs initializing. Replace the year numbers above by an i counter in a loop i = 1871, i =< 2005, i++ and you're good to go.

One thus computes the raw return of 1872 and other years from the raw data. One adds user bias to the raw return and computes a "new" index based on returns computed from the original, but with those returns biased. The biased index (or returns, however you've mechanized the code) is what is applied to the user's money.

I suggest compounding here because if you just increased each index year by 2%, the 1872 number would be 2% higher and so would the 1871 number. The rate of increase from 1871 to 1872 would not be 2% higher and we want it to be.

Does this make sense?
 
Wouldn't this defeat the point of FIREcalc? It's purely a historical sanity check. The historical results aren't just the function of annualized returns being high or low. The results are a function of the very specific historical sequences. The returns, the volatility, the duration of bull and bear cycles, the regression to the mean, etc.

Don't mess with it. The sort of tuning you're talking about is ideally done with MC simulators.
 
Don't mess with it. The sort of tuning you're talking about is ideally done with MC simulators.

If MC simulators were meaningful, biased or not biased, would the Trinity study ever have been done, and firecalc thereafter?
 
rodmail said:
If MC simulators were meaningful, biased or not biased, would the Trinity study ever have been done, and firecalc thereafter?

Of course MC simulators are meaningful.   The Trinity Study and FIREcalc simply take another approach to the question of portfolios surviving various conditions.

The question FIREcalc answers is "how would this portfolio have done historically?"   That's all it can tell you. [1]

MC simulators can also tell you how your portfolio would do through a bunch of alternative histories, generally much longer than 120 years each.    One approach isn't better or worse.   Just different, but FIREcalc is "pure" in that it uses historical sequences exclusively, so you would be diluting that if you tried to parameterize the results.

Taking historical "patterns" and fiddling with various parameters is exactly what MC simulators were designed to do.

Edit:
[1] Of course, it does much more than this.   It'll tell you how your portfolio would have done combined with a simple algorithmic withdrawal strategy.   But you get the idea -- don't mess with history!
 
Well, I guess the only rebuttal is that lots of tweaking is already in firecalc.

History can be abbreviated by choosing a year other than 1871 to start.

History didn't include mutual funds at 0.18% cost of investment. History had only full service brokers. So a test being performed with 0.18 expense is an approach not historically accurate. If it had been present in those years, might that have changed market behavior? Probably. So it is a tweak.

History is changed from the new firecalc vs the old one in that the final several years are truncated off the solution. Those years occurred, but they are not used.

I guess I don't understand the objection. The suggested mod is an option. I would certainly not want a default to be some skew of the historical data. If it's easy to code, why deny it to folks?
 
rodmail said:
I guess I don't understand the objection.  The suggested mod is an option.  I would certainly not want a default to be some skew of the historical data.  If it's easy to code, why deny it to folks?

I guess I never liked the swiss-army knife approach to anything.   Give me a wrench or give me a hammer, and keep your wrammer.  :)

Seriously, I think the mod would be meaningless.   Why pick on the return amplitude?   Why not stretch or contract the length of bull/bear cycles?   Why not move sequences around to see what happens when you don't get as much RTM?

I think a calc that let you do all sorts of these manipulations would be way cool, but I wouldn't recognize it as having any resemblence to FIREcalc or the study upon which it is based.
 
Seriously, I think the mod would be meaningless. Why pick on the return amplitude?

Because that is the matter under discussion. "Yeah, sure, you are getting 4.5% SWR with historical data, but suppose returns in the future aren't as good -- because the 20th Century was the American century. What happens if the next 30-40 yrs is weaker?"

Or . . . "Yeah, sure, you are getting 4.5% SWR but I think I could take more than that out and live a more lavish lifestyle because that 4.5% is based on a century where the investing public did not have access to the body of knowledge now present. That historical data is based on business that did not have the productivity of computers. I think the returns will be better in years to come. What happens to my SWR if they are better by X%?"

Why not stretch or contract the length of bull/bear cycles?

Because there is no algorithm, and certainly not one easy to code. You would need a Fast Fourier Transform applied to the data to extract the frequencies present in the waveform, filter the result in a low or high pass digital recursion and then imagine that you really did stretch or contract the wavelength of cycles that may or may not have even been present.

A simple bias of the returns up or down is a pretty reasonable answer to provide a person's idle question of what happens if returns are better or worse.

Why not move sequences around to see what happens when you don't get as much RTM?

Because it's harder to code, one would think, though that would be interesting.

I just offered a suggestion that I encounter often in discussion and one that I thought would be easy to implement. Shrug.
 
rodmail said:
"Yeah, sure, you are getting 4.5% SWR with historical data, but suppose returns in the future aren't as good -- because the 20th Century was the American century.  What happens if the next 30-40 yrs is weaker?"

Well, let me say that I'm not trying to impose my view on anybody.    If the feature were added, I might even play with it.   It's just that I'd have no clue what the result meant.

The problem is that we don't know the future, so any mod that tries to make things better or worse in some single dimension is really meaningless.    It gives us no new information.   We still don't know the future.   What do you think we'll learn by playing with the historical returns?

And, as I hinted at above, it's a slippery slope.   Is it weaker returns that would really hurt a potential retiree?    Ask any retiree, and they'll tell you that volatility and inflation are the real killers.   So, let's add some more features to play with those parameters.    The result is still meaningless.

My core point is that coming up with some *arbitrary* case that is worse than any existing worst-case is meaningless.    If you lower the historical returns, or increase the historical volatility, or change the historical RTM, these will all lead to a predictable result: a lower SWR.    But we can be pretty certain that our pretend scenario will have no resemblence whatsoever to our future, so it does us no good at all.

You raise a good point, though.    I would love a tool that let me model various future scenarios.   I'm not sure what I'd use the results for, but it'd be a fun video game.  :)
 
Oh, I'm totally okay with your points. They are valid questions.

The problem is that we don't know the future, so any mod that tries to make things better or worse in some single dimension is really meaningless. It gives us no new information. We still don't know the future. What do you think we'll learn by playing with the historical returns?


But this is why Sensitivity Analyses are done for all sorts of planning processes. When you don't know what the future holds, then it would be very useful to discover how sensitive some parameter is to another. The classic case is elasticity of demand, where one explores the impact of price on units sold. You move numbers around in test markets, gather data, plot the result. Intuition sometimes fools you. There have been cases where an increase in price increased units sold -- because a certain cachet was lent to the product.

But no matter. The point of Sensitivity Analysis in this context . . . is that maybe it takes a full 8% of market bias upward to move the % of successful runs at 4.5% from 95% to 96%. Would not such a discovery of profound insensitivity of success % to returns be useful to someone's planning? I would certainly get very conservative in my equity choices if I had data like that.

In statistics it is called ANOVA for analysis of variance. Dependent variables, by definition, vary as a function of independent variables. But the degree to which they do is not known until calculated.

Up to Dory, of course.
 
Have you already browsed some of the information at intercst's site?

http://www.retireearlyhomepage.com/safesum.html

He has varied several parameters to evaluate sensitivity.

Fun to play with, but here's what keeps sticking with me:  if you simply put your money under your mattress, the SWR for 30 years is 3.33% with no inflation adjustment (100%/30).   Instead of putting the money under your mattress, invest in something that merely keeps up with inflation and you'll do fine at that SWR.    Put the money in something like TIPS, and you get both the real returns + your nest egg depletion.

So, if you take volatility out of the equation, it's obvious that the SWR just scales linearly with returns.

If you want an interesting sensitivity analysis, I think some of the other things we discussed (volatility, cycle length, RTM amplitude, etc) are where the results start to get surprising.
 
If you want an interesting sensitivity analysis, I think some of the other things we discussed (volatility, cycle length, RTM amplitude, etc) are where the results start to get surprising.

Well, I'm a somewhat low level engineering executive trying to wind down to RE, so I have a bit of time now and then and would be willing to help -- perhaps algorithmically.

The items you've just mentioned -- there will be a challenge defining them for a codable algorithm.

For example, if you wanted to test the sensitivity of SWR success to future market volatility change, how do you do that. I suppose you take the data set of 135 values of the market index, compute a sloped mean, then the standard deviation around that upward sloping mean -- and then somehow, in entirely arbitrary fashion, change the values of the 135 points to yield a different value of 1 sigma. Yes, you can change the points to yield a 20% increase in sigma, but which points do you change to do that?

For cycle length, as I mentioned, you have to compute a Fast Fourier Transform to identify the sinusoidal frequencies in the data stream. Once you have them, you say, okay if I have lower frequencies the cycle wavelengths are longer, and vice versa. So to make the cycles be longer, you filter out the high frequencies, and then take that frequency domain data and inverse FFT it back to time domain.

Guy, that is not trivial coding. And when you're done you have to recognize that you forced a definition of sinusoidal behavior onto a datastream that may not be sinusoidal.

As for how powerfully a datastream tends to regress to a mean, well, that's a multi pole digital low pass filter. Coding multiple poles will be one hell of a lot of functions, with each additional pole commanded by a user declaring he wants regression to be more or less powerful. What the low pass filter will do is compress volatility to zero around either the mean of the overall data, or the mean of a slope of multi year market trend. Again, this is an almost undefinable algorithm.

I don't object to trying these things and I'd even be willing to help algorithmically. My original suggestion was not intended to focus on the superiority of future market performance bias wrt the past as the best candidate for SA. It was a combination. The combo was that having that feature available would be valuable to people who ask that idle question in discussions about SWR, but also it should be really easy to 1) define and 2) code.
 
That actually sounds like a cool approach.   I'd be surprised if somebody hadn't already run an FFT on the market gyrations.   Maybe I'll poke around to see what I can find.

If it were me, I'd try to fit the data to a Bezier spline, then you could rubberband the thing all over the place, including stretching about Y to shorten/lengthen cycles and stretching about X to vary return amplitude.    I have to admit, it's been 20+ years since I coded anything like that, but it sounds like fun.
 
Back
Top Bottom