For NYC home, yes because NYC is kind of one great big cesspool what with dogs crapping and peeing all over the place, not to mention other unpleasant sources of germs and "contaminants".
For weekend country house, yes, because don't want to track grass clippings, mud, deer poop, etc. in the house.
For other people's houses I always ask and come prepared to do whatever the host prefers.
For restaurants and bars, I tend to keep my shoes on unless its got a thatched roof and located on a sandy beach.
For the office, I sometimes remove my shoes when at my desk.
P.S. For contractors and repairmen we keep a supply of disposable shoe booties for them to use. When they come prepared with their own, its usually a good sign we're gonna like them.