For me the toughest part of the commute on the trains (Long Island Rail Road and either the NYC Subway or PATH trains) was simply having to switch from one rail system to the other, especially on the trip home when a delay on the Subway or PATH trains could cost me a LIRR connection. Living by a train schedule distorts one's perception of time in that if the subway or PATH train takes 5 minutes less time than expected, I don't get home any sooner but if it takes 5 minutes longer then I get home 25 minutes later.
Then there were the frequent delays on either train system, the frequent lack of alternative routes if there were a big problem, hot cars in the summer, cold cars in the winter, overcrowding due to shortened trains or on trains which lack the proper environmental controls. I often had to stand for 20, 30, 40 minutes at a time with no relief and sometimes on a hot car in the summer. I usually arrived at work exhausted and often a bit nauseous and always in a foul mood in the last 10 years of working.
But the more recent annoyance on the trains was not related to anything I described above. Instead, on the LIRR, it was the constant annoyance of rude and loud cell phone users who yakked about anythign and everything without caring about anyone around them. I wrote the LIRR several times asking them to establish "quiet cars" like Amtrak and MART (Maryland) have so those of us who wanted to have some peace and quiet could do so while the cell phone yakkers could yak away in their own cars. They never did this, and it was basically anarchy to get the phone yakkers to hush it up while the train crewmen were invisible and the onboard announcements to hush it up went ignored.
Driving from my place to Manhattan or to New Jersey during the rush hours was never a possibility because that would be much worse than the trains could ever be.
Ridding myself of the awful commute was by far the best part of ERing 3 years ago.