Great question. I thought about it all the time. My current situation is:
I like to own the best house my budget allows. $5M or less, $5000 sqfts, at least 5 bedrooms. I'm willing to put 20% of my money in this house because my family can enjoy it everyday. Certainly, it's a luxury item...