A little late to this thread...
Last year I spent about 15% of my liquid investments/9% of my total net worth on a second home that is a bay front condo. Why? (1) i had been looking forever for something that felt right that had the stellar views and was affordable (2) I loved it and it was a quality of life decision (3) I had a lot of cash sitting around doing nothing (4) My home that I also have of over 25 years is in a location where a major international company pulled and many amenities that were here are no longer here - but I love my house and am not ready to sell it (5) With the market highs and with cash sitting around, interest rates low, I felt it was a way to divert some $$, keep it out of the stock market, etc. I don't consider it "money in the bank" but it is "an asset"...sort of ..although I define an asset as something that makes you money not one you spend money on. A core of this asset will be preserved ...unless sea levels rise and take out the second floor, unless a hurricane slams it but I have flood insurance, unless, unless, unless. I am at the age I don't care nor do I worry about things like that anymore. There will always be reasons to keep you from doing something.
8 months later and I have never had one ounce of regret or buyers remorse. Bay front views as soon as I walk in the door. Best thing I have done in quite a while. Based on what you have written, I'd say...do what inspires you and your wife and what your hearts and gut instincts tell you. From the numbers you seemingly can afford it.