My DSO and I aren't retired quite yet, but we're those "millionaires next door", living in a modest, long-paid-off home, driving older cars, doing things ourselves whenever possible, etc. A few years ago we decided to increase our real estate holdings by buying a little get-away place while the housing market was down. We were able to pay cash for a great little place on a short sale.
One day we mentioned our second place to our neighbor, and she said, "Well, now you have two mortgages!" When my DSO politely informed her that everything was paid for, she gave us an incredulous look and clammed up. I got the feeling she thinks we both make minimum wage or something.
Living as she does in a bigger house than ours, hiring out her yard work and other routine maintenance, buying a new car every 5 or 6 years, and spending freely on new furniture, kitchen upgrades, and electronics, I suppose she can't imagine how anyone could own two places and be mortgage-free, especially the "poor" souls next door who can't even afford to hire a landscaper!