After agonizing over FIRE for 3 1/2 years, running the numbers until I realized I was the insane man doing the same thing over and over again expecting a different result.
IMO there are 3 areas in your life that you need to get comfort with before you will stop delaying FIRE.
1) psychological - you have to have the whatever that 'something' is that will replace your working. Hobbies, new career, professional student, volunteering, perpetual traveler, ...etc.
2) social - do you have a social network outside of work? If ALL (and I know some people where this is the case) of the people who you socialize with, hang out, drink beer, play cards, talk and laugh with, ...etc. are at work ... then you may want to rethink FIRE... If you have an 'outside of work' network then you're half way there.
3) financial - got to have enough to support whatever it is you want to do for the rest of your life (see item 1)
Once I figured out that there was a new (and somewhat scarey, being unknown) life after work and that I had things I needed to do before I 'kick', and that I could actually finance it with a degree of certainty, I figured out that I was going to be fine.
My 3 things may be a variation of what you all need to think about and consider to make yourself comfortable, .. but I don't think that it will (or should be) a bad boss or a stressful project or something that is makes the workplace worse.
Best of luck on your decision.