If you're not ready, you're not ready. And that's ok
Your FA tells you you can retire. Your calculations tell you you can retire. Most importantly, your wife tells you you can retire. If 2% will cover your cost of living, then I, an anonymous stranger from the interwebs, will echo their conclusion: You can retire.
Maybe the financial aspect is just an excuse. Sometimes our minds play tricks on us, dribbling on undeserved guilt and leading us to rationalize our way into behaviors we don't really want. Do you feel some desperate urgency to retire? I hope it's not because you've been reading this forum and think that unless you retire right away your life is wasted. Look up the abbreviation FOMO.
If you are tired of your j*b but unwilling to leave, citing financial fears which have no real merit, perhaps you should consider finding a different j*b. I know, easier said than done when you're over 50.
But may not be relevant, depending on the j*b you seek. High-paying, high-expertise, high-stress jobs can be tough to switch. If you're a surgeon or a pilot or a C-suite executive, it took a lot of effort for a lot of years plus a bit of luck to reach the top; a long campaign that you may not be up for repeating.
But suppose you aimed for a totally different occupation, one in which the real benefit is maximizing your fun instead of your paycheck. Two years ago, a friend of mine gave up his engineering career to follow his dream of live theater. His wife was on board, so he jumped in. His income probably dropped by 4X, but he has enough. Best of all, he's loving every minute of it. Good luck!