Study Questions Retirement Calcuators

[FIREd_2015]

In today's (2/20/16) print version of the WSJ - Study Questions Retirement Calculators

"...A new academic study of 36 such tools - many of them widely used - concludes that "in most cases, the available offerings are extremely misleading" and generally not helpful to consumers trying to figure out if they will have enough money to cover their expenses for the rest of their lives...The study was conducted by three researchers at Texas Tech University, including well-know financial adviser Harold Evensky...The researchers didn't identify the 11 tools that passed their test..."

Too late. I'm already out of the rat race...

New Study Questions Retirement Planning Calculators’ Accuracy - WSJ

 

[razztazz]

In today's (2/20/16) print version of the WSJ - Study Questions Retirement Calculators

"...A new academic study of 36 such tools - many of them widely used - concludes that "in most cases, the available offerings are extremely misleading" and generally not helpful to consumers trying to figure out if they will have enough money to cover their expenses for the rest of their lives...The study was conducted by three researchers at Texas Tech University, including well-know financial adviser Harold Evensky...The researchers didn't identify the 11 tools that passed their test..."

Too late. I'm already out of the rat race...

New Study Questions Retirement Planning Calculators’ Accuracy - WSJ

Wouldn't this undermine the entire concept of 401k's and related retirement veehickles?

 

[foxfirev5]

Too late here also. Now that I've given notice the last thing I need is to find out the calculators I've used are suspect. I figured it out for myself for the last 38 years, hopefully I'll still be able to do the math myself in the future.

 

[pb4uski]

That's interesting in that when I was deciding to retire I ran just about every calculator that I could find with as close to the same assumptions that I could since they each ask for different inputs and the results didn't seem that different to me.... all gave me different versions of a green light.

IIRC others have had similar experience. If that is so, how could this study conclude they are generally misleading and unuseful?

Edited to add: just found the study... need to read http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2732927

 

[FIREd_2015]

...
Edited to add: just found the study... need to read The Efficacy of Publically-Available Retirement Planning Tools by Taft Dorman, Barry S. Mulholland, Qianwen Bi, Harold Evensky :: SSRN

Thanks for finding the study!

 

[pb4uski]

Don't bother. This is an idiotic study. The authors seem to think that whether one is a smoker or non-smoker is important variable. Reading between the lines it seem they think the planning time horizon should be commensurate with health and since smoking has an impact on health then it is important to the retirement planning time horizon. However, there are many very old smokers out there who defy the odds.... including my 85 yo DM.

Am I misreading it? I concede that I just breezed through it but I'm not about to waste time on something so preposterous.

We find it interesting that “smoker vs. non-smoker” fell outside the “important” or higher importance classifications given the overwhelming evidence in empirical research of the strong negative impact smoking has on an individual’s longevity. In addition, Figure three shows that there is a noticeable gap in responses for the “smoker vs. non-smoker” and “personal health” variables among financial planning professionals. In aggregate, 43% of the respondents believe the variable “personal health” is “extremely important,” compared to 28% of the respondents believe it is “important,” one level from being classified as “not applicable.” In contrast, 45% of the respondents believe a tobacco use variable, such as smoker vs non-smoker, is “important” with only 29% of the respondents believing it is “extremely important.” Given the significant potential impact on mortality estimates, we are surprised that three times as many respondents believe a tobacco use variable is “not applicable” to building a retirement plan, as compared to respondents indicating “personal health” is “not applicable.” We wonder if the respondents considered the “personal health” variable a more inclusive variable.

 

[Tadpole]

Yea, you can look at all these calculators alone. I choose to believe the sigma/sqrt(N) sort of stuff. So if they all give me a good retirement prospectus, the final distribution of my retirement prospect is quite narrow (or so I choose to believe until any opposing real data comes along and then it will be a bit too late).

 

[Meadbh]

I quickly scanned the research paper. My first question was "what's the gold standard?". Of course, it would be outcomes, i.e. portfolio survival of a large cohort of retirees over decades, compared with predicted results from the calculators prior to retirement. Obviously, that would be impossible to study at this time.

The methodology the authors used was to review the inclusion of predictive data inputs in the various calculators. They focus on retiree-specific variables (age, income, assets are categories). Inflation, stock and bond return rates are also considered. Many of the calculators tested did not include variables that have been predictive in previous research and that the authors and a group of FAs considered important.

They then designed a specific marginal scenario and ran it through all the calculators. The results were wildly inconsistent. It is not clear how much of the inconsistency was due to variation in input variables or variation in calculator algorithms.

FireCalc was among the calculators tested, but is not listed in the final table, so it is not possible to review its performance in the study. Omission of data from the results makes me concerned.

The conclusion is that the design of publicly available calculators needs to be improved to avoid leading people astray. This makes sense to me. Retirement calculators are relatively recent innovations and will continue to evolve. In particular, their predictive values need to be validated with post hoc data. While it is not perfect, I do think this was a worthwhile study, if only to remind us not to encourage someone to ER just because "FireCalc says you're good to go".

 

[ejman]

During the mid to late 1990's when I first started thinking about ER I ran the calculators available at that time and I also got financial plans from a couple of independent Financial advisor’s at Ernst & Young and such, At the time they all said I was good to go and I ER'd end of 2002 at 52. Surprisingly, the most accurate forecast so far is one I ran in 1996 using QFP which back then had my liquid NW 20 years latter (in 2016) within 10% of what it actually is. I hope it continues to be accurate because it shows some very nice numbers for the 30th and 40th anniversaries...

 

[bclover]

So let's see....
1) financial advisers lead people astray
2) calculating tools lead people astray
3) most Americans are financially illiterate and don't know enough on their own.

Let's see if I save money, stress from above, If I don't save any money stress that I can never retire.


I'm a newbie at this and in the 16 months I've been preparing myself (or at least attempting to) I think I've figured it out. Just like the latest food warnings, no matter what I do, if I wait long enough some one will come along and tell me to start worrying because it's wrong.

Glad I'm going to Vegas next month, think I'll upgrade my hotel to a suite and start enjoying spending. lol, at least when someone says it's wrong I can say I had a good time.

Jeez

 

[pb4uski]

During the mid to late 1990's when I first started thinking about ER I ran the calculators available at that time and I also got financial plans from a couple of independent Financial advisor’s at Ernst & Young and such, At the time they all said I was good to go and I ER'd end of 2002 at 52. Surprisingly, the most accurate forecast so far is one I ran in 1996 using QFP which back then had my liquid NW 20 years latter (in 2016) within 10% of what it actually is. I hope it continues to be accurate because it shows some very nice numbers for the 30th and 40th anniversaries...

Your post caused me to go back and look at my actual to expected. My end of 2015 portfolio value is 17% higher than what I expected when I retired at the end of 2011, totally due to better than expected investment returns (9% vs 5%).

 

[rodi]

This study definitely seemed biased or incomplete. As pb4uski pointed out - they seemed obsessed with tobacco use and it's impact on longevity... But most calculators don't predict how long you'll live. For example Firecalc asks you to input the length of retirement (putting the burden on the user... )

I would have liked to see the scenario data and results from the 36 calculators. It wasn't there.

Folks on this board tend to be fairly well educated about the possible pitfalls of most retirement plans and take steps to mitigate - or at least consider the steps they might take if needed. I suspect that the vast majority of folks not on this board do not have the knowledge, understanding, or really care that much about these nuances... and if their simple calculator or 4% rule of thumb says they are good to go, they figure they are good to go.

 

[Car-Guy]

I "pity the fool" who would retire solely based on one or more of these on-line calculators (including FIREcal). I think many are interesting and fun to use, however, I actually trusted my custom spreadsheets and common sense a lot more to help me make my decision to retire.

 

[alaska55]

I hope you are being sarcastic and not calling a lot of people fools.


I "pity the fool" who would retire solely based on one or more of these on-line calculators (including FIREcal).

 

[Midpack]

That was pretty vague and therefore useless. Some retirement calculators are misleading (definitely true, they're pretty easy to recognize), so you can't trust any of them. Thanks for the thoughtful insight.

Some Company ABC retirement calculators are black boxes that don't reveal underlying assumptions and their purpose is to sell you Company ABC investments and/or financial advice - really, isn't that shocking! Anyone who plugs in a few numbers without any idea what the underlying assumptions and calculations are, and believes the output is an idiot. Caveat emptor.

They list a bunch of calculators in Appendix A, but never provide any indication re: how useful any of them are/aren't. The results are highly variable, so you can't trust any of them. That would be like Consumer Reports testing all automobiles, finding some more reliable than others, and concluding you can't rely on cars. Huh?

And they say you can't rely on financial advisors either.

So work until you drop? Or take up smoking?

I guess if your a smoker you probably have enough, and if you're not you'll probably outlive your nest egg.

The paper wasn't at all value added for me. YMMV

 

[Senator]

I had to re-read the study. A lot what was said had to do with user input. Some people did not know how to input the data, some calculators did not allow enough data, and smokers die faster. I also got the impression they were advocating for pensions.

Conclusion, There is no 100% way to predict what will actually happen.

At the end is a great list of calculators all with hyperlinks. I tried out a few, and they are basically the same. If that is all you get out of this study, it is worth it.

You can use historical results (FireCalc), you can use Monte Carlo, straight-line, and some sort of blend. There really is not much else.

I tried numerous calculators over the past few years, and none can tell me with 100% certainly I will be OK. They might give a statistical probability of 100%, but no one can ever say until the absolute end, "It worked". It is a leap of faith, similar to getting in your car and expecting to arrive back at home in one piece.

 

[pb4uski]

I had not noticed the hyperlinks.... most useful part of the document IMO.

I did notice that they put in only partial information on their test scenario so readers could not try to replicate their results.

 

[sengsational]

The conclusion is that the design of publicly available calculators needs to be improved to avoid leading people astray. .

Some Company ABC retirement calculators are black boxes that don't reveal underlying assumptions and their purpose is to sell you Company ABC investments and/or financial advice - really, isn't that shocking!

Even if there's pages of assumptions associated with the calculator, if it just spits out "an answer", I'm going to be at least a bit skeptical. It would be better if I could download balances and flows year by year to see what it actually did. I'm sure there are calculators out there that have huge bugs that will never be found since we have only a black box / can only see the input and output, nothing in between.

 

[Midpack]

Even if there's pages of assumptions associated with the calculator, if it just spits out "an answer", I'm going to be at least a bit skeptical. It would be better if I could download balances and flows year by year to see what it actually did. I'm sure there are calculators out there that have huge bugs that will never be found since we have only a black box / can only see the input and output, nothing in between.

FIRECALC will return year by year results FWIW. Though it's still difficult to recreate what happens in the black box.

There is no perfect retirement calculator or advisor, and never will be. You look at historical results, and make your own judgement re: how that will fare over the course of your retirement. It's up to you to factor in input variables, probabilities, etc. GIGO

 

[sengsational]

FIRECALC will return year by year results FWIW. Though it's still difficult to recreate what happens in the black box.

I think there's a spreadsheet 'out there' that has the same premise as FIRECalc and can be used to get at least roughly the same result. I've made a spreadsheet that validates i-orp output, given what it decided to do, flow-wise. So yeah, there are a few that lend themselves to sanity checking. I trust those more than a black box.

 

[pb4uski]

Even if there's pages of assumptions associated with the calculator, if it just spits out "an answer", I'm going to be at least a bit skeptical. It would be better if I could download balances and flows year by year to see what it actually did. I'm sure there are calculators out there that have huge bugs that will never be found since we have only a black box / can only see the input and output, nothing in between.

That is why I ran numerous calculators before I retired. They all gave me a green light... from light green to solid green. That is one of the reasons why I like Quicken Lifetime Planner... I can see the year by year projections and recalculate each item. I concede that QLP is deterministic so it does not address sequence of returns risk and that is why I use Firecal and others as supplements to assess that risk.

 

[Gone4Good]

I think the paper is more worthwhile than the WSJ article. And by that I don't mean it is necessarily a worthwhile read for members of ER.org. Most of us won't learn anything new there.

But that doesn't mean it isn't a good and necessary critique of the shortcomings of web-based retirement planning calculators - particularly as used by the average Joe and Jane.

For example, this seems like a worthwhile thing for researchers to highlight, if not shout from the rooftops . . .

Providers of publically-available retirement planning tools have the opportunity to exploit naïve users through information asymmetry (see Akerlof, 1970). Many tools market themselves as sufficient to answer the most common retirement question: how much do I need to save for retirement? However, tool inputs and default settings are established to guide households into purchasing financial products or speaking to a financial professional who may be paid for selling proprietary products. Due to their design, allowing tool providers to exploit the information asymmetry suggested byAkerlof, households may view tool results as retirement advice, without the understanding that the tools provide directed “advice” that benefits the vendor, not the investor.

And regarding the complaint here that the longevity issues raised by the researchers (e.g. smoking, etc) are somehow overblown or unimportant, they pre-rebutted that complaint in the text . . .

A MetLife (2008) survey finds that about 60% of respondents underestimate the likelihood of a person living to age 85. If a tool relies on the typical resource for determining life expectancy in the U.S., the Social Security Administration’s Period Life Tables 10 for 2011 (see SSA, 2011), then there is a 50% chance the individual will surpass their indicated life expectancy

Again, not necessarily useful to those of us here. I personally plan to live forever so my retirement horizon is fixed at ∞. But it appears most folks massively underestimate their possible longevity. And many calculators either leave that estimate up to the user (FireCalc) or default to a 30 year window (FireCalc).

It's easy to see how the average person can get misleading results from these tools. It's no bad thing to point that out and highlight possible areas for improvement.

 

[haha]

A MetLife (2008) survey finds that about 60% of respondents underestimate the likelihood of a person living to age 85. If a tool relies on the typical resource for determining life expectancy in the U.S., the Social Security Administration’s Period Life Tables 10 for 2011 (see SSA, 2011), then there is a 50% chance the individual will surpass their indicated life expectancy

I believe that this highlights a common bias.-People tend to want "the answer", not the surrounding information that would be necessary to correctly interpret the findings.

It is a form of figure/ground bias. Often more meaningful information is in the ground than the figure that is being highlighted. Of course "life expectancy" is a median, and it also assumes that mortality rates are stable from whenever the data was collected, and also that the individual who is trying to use the information is drawn from the same sample, or in Bayesian terms that one has chosen a relevant reference group.

But nothing to be done for others, it is all most of us can do to make our own good choices.

Ha

 

[marko]

After spending far too many hours going through such calculators and coming out with about the same result, I got a NEW retirement calculator on my phone.

I plug in my portfolio total and enter "X 4%" and I get my number! It's amazing.

As has been stated, all such calculators can only give you a 'windage', a sort of level of the water. There's just so many variables and as "life happens" over the next 20-30 years one must adjust.

It's fun to get regular confirmation and to do that double-check every so often but I surely hope no one takes any of these calculators to the n-th decimal believing it's rock solid.

 

[Options]

This book made me look at all such new studies with a healthy dose of skepticism:

Wrong: Why experts* keep failing us--and how to know when not to trust them *Scientists, finance wizards, doctors, relationship gurus, celebrity CEOs, ... consultants, health officials and more: David H. Freedman: 9780316023788: Amazon.com: Books

Studies presumably provide the answers. In examining hundreds of these studies, Ioannidas did indeed spot a pattern--a disturbing one. When a study was published, often it was only a matter of months, and at most a few years before other studies came out to either fully refute the findings or declare that the results were "exaggerated" in the sense that later papers revealed significantly lesser benefits to the treatment under study. Results that held up were out-weighted by by results destined to be labeled "never mind".

and

The facts suggest that for many, if not the majority, of fields, the majority of published studies are likely to be wrong," he says. Probably, he adds, "the vast majority."