Study Questions Retirement Calcuators

[pb4uski]

Quote:

Originally Posted by sengsational Even if there's pages of assumptions associated with the calculator, if it just spits out "an answer", I'm going to be at least a bit skeptical. It would be better if I could download balances and flows year by year to see what it actually did. I'm sure there are calculators out there that have huge bugs that will never be found since we have only a black box / can only see the input and output, nothing in between.

That is why I ran numerous calculators before I retired. They all gave me a green light... from light green to solid green. That is one of the reasons why I like Quicken Lifetime Planner... I can see the year by year projections and recalculate each item. I concede that QLP is deterministic so it does not address sequence of returns risk and that is why I use Firecal and others as supplements to assess that risk.

[Gone4Good]

I think the paper is more worthwhile than the WSJ article. And by that I don't mean it is necessarily a worthwhile read for members of ER.org. Most of us won't learn anything new there.

But that doesn't mean it isn't a good and necessary critique of the shortcomings of web-based retirement planning calculators - particularly as used by the average Joe and Jane.

For example, this seems like a worthwhile thing for researchers to highlight, if not shout from the rooftops . . .

Quote:

Providers of publically-available retirement planning tools have the opportunity to exploit naïve users through information asymmetry (see Akerlof, 1970). Many tools market themselves as sufficient to answer the most common retirement question: how much do I need to save for retirement? However, tool inputs and default settings are established to guide households into purchasing financial products or speaking to a financial professional who may be paid for selling proprietary products. Due to their design, allowing tool providers to exploit the information asymmetry suggested byAkerlof, households may view tool results as retirement advice, without the understanding that the tools provide directed “advice” that benefits the vendor, not the investor.

And regarding the complaint here that the longevity issues raised by the researchers (e.g. smoking, etc) are somehow overblown or unimportant, they pre-rebutted that complaint in the text . . .

Quote:

A MetLife (2008) survey finds that about 60% of respondents underestimate the likelihood of a person living to age 85. If a tool relies on the typical resource for determining life expectancy in the U.S., the Social Security Administration’s Period Life Tables 10 for 2011 (see SSA, 2011), then there is a 50% chance the individual will surpass their indicated life expectancy

Again, not necessarily useful to those of us here. I personally plan to live forever so my retirement horizon is fixed at ∞. But it appears most folks massively underestimate their possible longevity. And many calculators either leave that estimate up to the user (FireCalc) or default to a 30 year window (FireCalc).

It's easy to see how the average person can get misleading results from these tools. It's no bad thing to point that out and highlight possible areas for improvement.

[haha]

Quote:

Originally Posted by Gone4Good A MetLife (2008) survey finds that about 60% of respondents underestimate the likelihood of a person living to age 85. If a tool relies on the typical resource for determining life expectancy in the U.S., the Social Security Administration’s Period Life Tables 10 for 2011 (see SSA, 2011), then there is a 50% chance the individual will surpass their indicated life expectancy

I believe that this highlights a common bias.-People tend to want "the answer", not the surrounding information that would be necessary to correctly interpret the findings.

It is a form of figure/ground bias. Often more meaningful information is in the ground than the figure that is being highlighted. Of course "life expectancy" is a median, and it also assumes that mortality rates are stable from whenever the data was collected, and also that the individual who is trying to use the information is drawn from the same sample, or in Bayesian terms that one has chosen a relevant reference group.

But nothing to be done for others, it is all most of us can do to make our own good choices.

Ha

[marko]

After spending far too many hours going through such calculators and coming out with about the same result, I got a NEW retirement calculator on my phone.

I plug in my portfolio total and enter "X 4%" and I get my number! It's amazing.

As has been stated, all such calculators can only give you a 'windage', a sort of level of the water. There's just so many variables and as "life happens" over the next 20-30 years one must adjust.

It's fun to get regular confirmation and to do that double-check every so often but I surely hope no one takes any of these calculators to the n-th decimal believing it's rock solid.

[Options]

This book made me look at all such new studies with a healthy dose of skepticism:

Wrong: Why experts* keep failing us--and how to know when not to trust them *Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, ... consultants, health officials and more: David H. Freedman: 9780316023788: Amazon.com: Books

Quote:

Studies presumably provide the answers. In examining hundreds of these studies, Ioannidas did indeed spot a pattern--a disturbing one. When a study was published, often it was only a matter of months, and at most a few years before other studies came out to either fully refute the findings or declare that the results were "exaggerated" in the sense that later papers revealed significantly lesser benefits to the treatment under study. Results that held up were out-weighted by by results destined to be labeled "never mind".

and

Quote:

The facts suggest that for many, if not the majority, of fields, the majority of published studies are likely to be wrong," he says. Probably, he adds, "the vast majority."

[Gone4Good]

Quote:

Originally Posted by Options This book made me look at all such new studies with a healthy dose of skepticism:

I think that's why you don't see anyone outside the media pay too much attention to any one study. Often times it takes multiple studies done over numerous years with a variety of different methodologies to arrive at a consensus conclusion among experts. And that conclusion may not even be reflected in a single study, but represent the cumulative findings of many.

Of course that is all too painstaking for the general public. As Ha says, "just gimme an answer" . . . especially one that confirms what I already believe.

[OrcasIslandBound]

There are some "good" calculators out there. I think FireCalc is one, Fidelity RIP tool is another. Consumer reports points out that the better calculators use historical data combined with Monte Carlo simulations to get the results of predicting success rates. However, the implied assumption is that history will repeat itself. Some history examples violate these assumptions however. Example, when a wheel barrow full of money wouldn't buy a loaf of bread in Germany during the 30's. If this happens again, we'll all be focused on what we happen to have in the cupboard and car.

Sent from my Nexus 4 using Early Retirement Forum mobile app

[rdy2go]

I had a friend in college that was finishing up his doctorate in statistics. he earned pocket money by doing the statistical work for other grad students for their thesis or dissertation work. He would ask them "what do you want the numbers to prove?" and would make the data fit whatever angle they were pursuing, knowing full well that no one would understand the statistical work presented. He could make the data say whatever he wanted it to say and no one was the wiser. He went on to work for NASA by the way.

[aggie76]

I think I found a simple solution to the issues presented in the study. Whenever I use the MANY MANY calculators I always plug in the absolute worse case ( within reason) inputs. I have inflation at 3%, I input portfolio return at 4%, life expect at age 95 for both of us ( not at ALL likely)....etc.... I figure that IF the various calcs give me a high probability under those conditions I should be relatively safe under historical norms.
Of course someone always brings up the Mad Max or the Weimer Rep scenarios....Simple...All that will count after the zombie or money wheelbarrows events will be my GUNS AND AMMO...I will save the last 2 rounds for my and wife and I as in..who the hell in their right mind wants to live in that world !!

[Midpack]

Quote:

Originally Posted by Gone4Good I think that's why you don't see anyone outside the media pay too much attention to any one study. Often times it takes multiple studies done over numerous years with a variety of different methodologies to arrive at a consensus conclusion among experts. And that conclusion may not even be reflected in a single study, but represent the cumulative findings of many. Of course that is all too painstaking for the general public. As Ha says, "just gimme an answer" . . . especially one that confirms what I already believe.

+1.

That's why you have to take many media reports with a grain of salt, and why you see so many findings seemingly reversed in time. Most media outlets are more focused on getting your attention, than reporting news...

And some here seem to take that approach too, posting an article or study that provokes replies...but no substantial or new information.

[Andy2]

There is no perfect answer to this:
Most calculators provided by financial institutions are biased towards "encouraging" people to save more than they might need for longer as that suits their own financial interests.
Spreadsheets can be really good but are maintenance intensive and a bad formula can be invisible and skew the results
I personally do not like Monte Carlo simulations...

In my opinion you need to develop a plan that works for you based on a combination of the best tools you can find..

[Big_Hitter]

Quote:

Originally Posted by Andy2 I personally do not like Monte Carlo simulations...

then how do you define "accuracy" with regard to retirement planning?

[papadad111]

Quote:

Originally Posted by Meadbh I quickly scanned the research paper. My first question was "what's the gold standard?". Of course, it would be outcomes, i.e. portfolio survival of a large cohort of retirees over decades, compared with predicted results from the calculators prior to retirement. Obviously, that would be impossible to study at this time. The methodology the authors used was to review the inclusion of predictive data inputs in the various calculators. They focus on retiree-specific variables (age, income, assets are categories). Inflation, stock and bond return rates are also considered. Many of the calculators tested did not include variables that have been predictive in previous research and that the authors and a group of FAs considered important. They then designed a specific marginal scenario and ran it through all the calculators. The results were wildly inconsistent. It is not clear how much of the inconsistency was due to variation in input variables or variation in calculator algorithms. FireCalc was among the calculators tested, but is not listed in the final table, so it is not possible to review its performance in the study. Omission of data from the results makes me concerned. The conclusion is that the design of publicly available calculators needs to be improved to avoid leading people astray. This makes sense to me. Retirement calculators are relatively recent innovations and will continue to evolve. In particular, their predictive values need to be validated with post hoc data. While it is not perfect, I do think this was a worthwhile study, if only to remind us not to encourage someone to ER just because "FireCalc says you're good to go".

Sadly no calculator can predict the future. Not future returns. Not future life expectancy.

If one is conservative in their WR, stay in the low 3 percent range and holds an AA on order of 80/20, then they should be ok by all calculators.

If not, then the surprise would not have been predictive by any of the calculators anyway. (Enter asteroid...).

It's really the ones on the very margin that are at risk and then it really becomes a game of craps. Throwing the dice to see what happens. There will be some winners and some losers based on happenstance. That's life. That's Lady Luck.

The entire concept of retirement is but a generation...maybe two generations old. It's all new. Makes sense there are no well dialed calculators to predict the future.

Flux capacitor, anyone?

[MichaelB]

Quote:

Originally Posted by papadad111 Sadly no calculator can predict the future. Not future returns. Not future life expectancy. If one is conservative in their WR, stay in the low 3 percent range and holds an AA on order of 80/20, then they should be ok by all calculators.

The usefulness of a calculator is not because it predicts the future, it is in the way it frames and models the changes and flows of assets, income and expenses over time. The paper makes this point, but in a rather muddled way. Still, as Meadbh points out, their two recommendations are sensible. First, smoking as a proxy for overall health, and second, focusing on default parameters, both could make calculators more useful.

FIRECalc modeling shows that (when in the withdrawal phase) increasing the equity allocation above 40% - 50% does not significantly improve the probability of portfolio survival. Lowering the withdrawal rate from 4% to 3% does, but only slightly.

One characteristic of a useful calculator would be to measure and project the range of outcomes and impact on portfolio survival when changing these two parameters.

[Gone4Good]

Quote:

Originally Posted by MichaelB The usefulness of a calculator is not because it predicts the future, it is in the way it frames and models the changes and flows of assets, income and expenses over time.

I think that is precisely right.

As a first pass you want a model to tell you whether you're plan is reasonable. But that is really only the first pass. A good calculator helps you think about and investigate what success actually means.

That is why FireCalc is so useful. It's not a black box. We know the assumptions. We can debate whether those assumptions are reasonable. And if we come to the conclusion that they're not, we can adjust our strategy to account for what we think of as the flaws in the model.

Another huge benefit of a calculator like FireCalc is that you can dig into the results. It's not enough for me to know that my plan was Successful when success is defined as having $1 left when I die.

FireCalc gave me the ability to follow one of the adverse market scenarios. And by doing that I got a good sense that what the model saw as a "successful" scenario (living on less than a quarter of your original assets for a decade or more, all the while bumping against financial ruin and hoping to die on schedule) wasn't even close to how I'd want to actually live.

Most calculators just give you a number. That's not nearly enough.

[Gone4Good]

Quote:

Originally Posted by MichaelB FIRECalc modeling shows that (when in the withdrawal phase) increasing the equity allocation above 40% - 50% does not significantly improve the probability of portfolio survival. Lowering the withdrawal rate from 4% to 3% does, but only slightly.

If you only look at portfolio survival that definitely seems true. Moving from a 4% withdrawal over 30 years with a 60/40 AA to a 3% withdrawal increases portfolio survival from 95% to 100%.

That doesn't look like a big deal. But that's only because we're using a statistic that's capped out at 100% - any improvements beyond that are completely missed. If we look at the ending portfolio balances we get a much better sense of what that 100bp difference in withdrawal rate really means.

With a 4% WR on a $1MM portfolio the worst-case ending balance bottoms out at ($272K). At 3% WR my balance bottoms out at $561K. That's a huge difference. One strategy results in potential ruin while the other has a downside scenario where I still have more than half my original assets.

But that's not all. If we tell FireCalc that we want to maintain a $560K balance we learn that a 4% withdrawal fails to do so 45% of the time.

In other words, a 4% withdrawal strategy results in a portfolio that declines below the worst case scenario of a 3% withdrawal nearly half the time. That's an enormous difference and one that a simple "portfolio survival" metric completely misses.

To bring things back on-topic, the ability to do this kind of analysis is why FireCalc is such a great tool.

[samclem]

Quote:

Originally Posted by Gone4Good FireCalc gave me the ability to follow one of the adverse market scenarios. And by doing that I got a good sense that what the model saw as a "successful" scenario (living on less than a quarter of your original assets, bumping against financial ruin, hoping to die on schedule) wasn't even close to how I'd want to actually live.

+1. I try to imagine what it would be like, emotionally, to ride some of those "successful" lines in real life and with no ability to "peek ahead" and see that the line eventually takes an upward turn. It would be very uncomfortable.

[hnzw_rui]

Quote:

Originally Posted by papadad111 The entire concept of retirement is but a generation...maybe two generations old. It's all new.

Hmm, my grandparents have been retired for as long as I can remember. I'm Asian so for me, multi-generational households are the norm. Of course in our case, the middle generation live in the family home and fund their parents retirement while the older generation are built-in babysitters so both husband and wife can go to work.

[photoguy]

Quote:

Originally Posted by papadad111 Sadly no calculator can predict the future. Not future returns. Not future life expectancy.

Retirement calculators are all about predicting the future, either directly or indirectly. If they weren't relevant to our future, we wouldn't care about them. There are also tons of other statistical models that can predict future equity returns and life expectancy.

Now if by prediction, you mean providing an accurate point estimate like saying your $1M portfolio will be worth 1.56M in ten years. then yes retirement calculators have nowhere near this level of power and would be useless at this task.

However a calculator can do things like provide an expected outcome and the dispersion of results. E.g. on average the portfolio value is 1.5x but could vary as much as 0.5x to 2.5x. As others have noted, perhaps the most important thing about interpreting the outputs is understanding how much variance there can be in results.

[sengsational]

Quote:

Originally Posted by MichaelB One characteristic of a useful calculator would be to measure and project the range of outcomes and impact on portfolio survival when changing these two parameters.

To address the inability of some calculators to perform a Monte-Carlo simulation, one can do 'sensitivity analysis' like this as an alternative. Monte Carlo tries a bunch of random things, and typically many of them don't make any sense (the market is not a uniform distribution, which is what is used much of the time). If you run many scenarios with various ranges of inputs, you can get a pretty good idea of what the range of possibilities would be.