I can totally relate. In terms of job stress, it's probably not you. What I've found is that over the years, my work has become "easier" because I have accumulated a massive base of skills, experience, and networks from which to draw upon. But, the bad news is that I've also gained increasing levels of responsibility. When the buck stops at your door, there is a different kind of stress. And that stress is less about number of hours worked, and more about the invasive mindshare the work occupies.
I can't just turn it off when I come home or even go to sleep, and thanks to all the modern telecommunications we now enjoy, and the global nature of business, all the folks relying on me expect to be able to reach me anywhere, anytime, anyplace. And I don't know about you, but I feel like the business world has devolved into a last minute everything culture. So, now everything is a crisis, everything is 11th hour, and everything is a make or break scenario. After about the 10th urgent disaster of the week, my nerves are usually frayed.
It doesn't need to be this way - we (i.e. our society) has just decided somehow that this is how we define "productivity" - just use you up until you're a spent shell, then discard you for the newer model. Of course, as you get deeper into FI, that will help relieve some of the stress. The worst feeling in the world is to feel trapped in a bad situation.
I too worry that I'll spend too much time worrying in retirement - just this morning DW was pondering "hey hon, what if we bought a condo/co-op second home in retirement, all-cash, cause we won't have any income" me, blood pressure rising, vision narrowing, seeing more $$$ sucked from diversified liquid assets into illiquid single asset - breath in, breath out, I told myself, she's just pondering, let her ponder, don't say anything, don't move, don't flinch, her mind will wander on to something else, yup, ok, coast is clear now. I'm going to need to get used to this.
