Sounds sort of like when I stumbled into a low-stress well-paying job after retiring the first time. For as long as things went well it was fine, and when things went south I had the option to leave. And I did.
You're in the same position. Go for it.
I heard the call to do nothing. So I answered it.