I think we often have rose-colored-glasses on when looking in the rear view mirror.
There were things I liked about my job. Solving problems, implementing cool stuff, collaborating with cool people. But the things I didn't like come with it, especially in IT:
On call. Weekend Installs. Joining late night bridge calls for outages. Reporting to Upper Seniors on why there was that outage. Super early morning or late night calls with the offshore team. Vendor contract negotiations. Management who wanted re-negotiations every year to get more for less when it was always less for less. Ever dwindling onshore support for more and more offshore.
The BS, the reporting, the budget fights. Running on stuff that needed upgrading, spending a month putting together a business case that had to be redone 17x for your boss, then, getting told, no money, again. Business partners who never wanted to compromise on scope, schedule, or budget and then never showed up to help test, and invented workarounds in production which you found out about 3 years later when you updated code and found broke their stupid workaround that they never told you about and now they've escalated to high levels before even telling you...
I mean if none of that is making you shudder, go back!