I enjoy traveling... but I hate to travel! Mostly I dislike dealing with the Aeropuerto... it is stressful.
I have also found foreign train stations and subways to be a bit confusing (until we get our bearings).
But most of my leisure travel stress has been related to squeezing a vacation into a week or two. I always seem to have had a heightened sense of urgency about getting there on time and meeting a tight schedule.... one delay or problem could ruin an expensive vacation.
When I FIRE, we will travel and I am sure there will still be some stress, but since we will not be artificially constrained... and more flexible, I hope to relax, consider it to be part of the adventure, and roll with it.
Since I do most of the planning and arranging, DW does not have to deal with that part so it does not bother her as much. Her problem is that she gets a little anxious flying... but she handles it fine.