The bottom line is that rate of return and the alternative (I guess) use of the home as well as it being an asset.
Historical returns run around 8% before taxes, your tax rate will determine the "real" return. Twenty-five percent is the average, so you'll be seeing about 6-ish%. On average. Over long periods of time. There have been several ten and twenty year periods where the real rate of return was zero. Even at the historical rate of return compared to mortgage rates, over 30 year spans, you might eke out a percent or two advantage.
Your real rate of return on paying off the mortgage is the mortgage rate. Guaranteed. And I'm unaware of any guaranteed investment vehicles that carry a 5.something% rate of return. If someone knows of one, let me know. My money heads there the next day.
As far the future, many experts agree that future return rates will be lower than they have been. If you believe experts. One says 3-3.5% "real" return, another says 6-9 before taxes, or 4-7 after, depending on your situation.
Factoring in the mortgage deduction is also worthwhile, although many folks including me and my parents, pay no taxes so a deduction would be worthless.
I'm also afraid I have to apply some rationality to the "irrational" fear of losing ones home for the folks who love history lessons when they're favorable, but ignore them when they're not. Many times in our history, the economy got bad enough that a lot of people DID lose their homes. However I'm not even remotely concerned about that. What I *am* concerned about is sustained high withdrawal rates during extended downswings in the market. As mentioned about 97 times now, I can reduce my withdrawal rate to about 8-9k a year for a couple of years with only a few luxuries absent, and to $12k with very little off of my current lifestyle.
With a mortgage, I cant do that. So having a declining portfolio that started off 200k larger and having to take large bites of principal out of it to pay the debt served would kinda suck in a 5-7 year bear market. You know, the kind you frequently see in our history lessons when equities are highly overvalued. Like they are now.
So unless you have a really small portfolio (under 500k), are paying less than 4.5% on the mortgage now, or are past the halfway point in payments, it bears considerable thought to pay it off.
And again, my homes value can go down 100% and I can still live in it for 50 years. My stocks and bonds go down 100% and I have wallpaper.
If its just too complex and wigs you out, but it still seems attractive, then take dollars directly from the fixed income portion of your portfolio and apply it to the mortgage. The mortgage is more guaranteed and pays a higher rate than anything of comparable risk in the fixed income area and is immune to interest rate risk. An unbeatable combo, which is probably why banks get rich on mortgages
I'm getting that feeling of Vuja-De. Thats where you get that feeling that you've done this before and never want to do it again.