I think your plan is far from crazy -- having just finished a transition quite similar to what you're contemplating, I recommend you take the plunge.
A few years ago, my wife and I were in a similar boat. I was (and still am, barely) in my 20s, in a job earning good $$. My wife was in a good career. Not as much money, but the benefits were good. The problem is that our jobs were consuming every ounce of energy we had, and I simply couldn't see myself doing that for the 20-25 years I estimated I needed to FIRE. In fact, back then I used to frequent these boards, posting from time to time. But I was so deflated when I realized that I had another 20-25 years in a job I didn't enjoy. Ok, I'll be honest -- a job (and career track) I hated.
So I gave up my FIRE dreams. I refocused my energy from FIRE to finding a vocation that made me happy every day. I did that and haven't looked back.
The funny thing is -- our finances now are in a better position than they were previously in my higher paying job. My wife stays home with the wee one, and we have more opportunities to travel. The benefits in my new job are phenomenal. And here's the kicker -- when I sit down now (2-3 years after I went through my discouraged phase) and look at the retirement calcs, I realize that I am in a better position to FIRE safely and comfortable at 50 than I was in my prior higher paying job. Sure, it's still 20 years away. But I actually look forward to work most days, and I think the 20 year journey will be far more enjoyable.
I still see myself retiring early some day. But now I don't see myself being miserable during the working years.
Long story short -- do what makes you happy and the rest will fall into place.
Sorry for the rant, but the original post really hit home with me.