This would be an interesting topic for another thread - how much people feel they have sacrificed to achieve ER. It has probably already been the subject of a thread, I imagine. I don't feel that I sacrificed much at all, as I'm naturally wired to save, and not consume excessively....
We are basically wired like you, and from very early on had a plan to retire relatively early to travel. Others sure thought we were weird though. DW was the only doc who drove a cheaper car than most of her staff. I drove a cheaper car than any other lawyer that I worked with (and most of support staff). And, our conflicting schedules limited our travel (which is why we wanted to retire....).
But, all the kids got challenging schools and highly selective colleges without loans, and our annual week of travel as a couple was generally a pretty nice dive trip. We really didn't feel like we sacrificed at the time--although we sure do enjoy going out to eat every week now, not feeling compelled to always buy generic, and not characterizng grilled Chuck roast as "steak."
LBYM is the key--whether "sacrificial" or not.