As far as intrinsic factors go, others have weighed in so I won't comment.
The word "investing" in your opening sentence caught my attention. I've been running numbers lately because we are selling a second home we bought almost 20 years ago (bought it's replacement two years ago-it took a long time to sell this one).
We are selling for a huge gain. That's good, although it will come with a big tax bill. However, the price appreciation just barely covered our expenses from owning and maintaining the property over the years. The tax bill will put us in the hole a few hundred thousand.
We were there roughly eight to ten weeks every summer, with very infrequent long weekend trips. It is 1,000 miles from our primary residence, fairly remote, and in a very high cost area. We did not rent it.
So while it might look like a good "investment" it depends on how you look at the numbers. Would we have had some of those costs anyway if we only stayed in the other house all year? Some, but not a lot. Taxes, insurance,
interest, maintenance are all extra. Maybe a bit of savings on utilities by not living in the primary house all year.
Would we have made a different decision knowing these numbers in advance? No, but a big gain in appreciation was offset by expenses to own over the years. Clearly a lifestyle decision that has a cost to it.