…
I don't look down on anyone because of their height (but then I'm not super tall at the old male average of 5'8" -I think it's crept up a bit) and doubt anyone looks "down" on you in the negative sense. The only realm I think it might matter is in dating. For me, being on the market, there are a lot of women out there looking for taller men -even if they are shorter than I am. Every once in a while a taller friend will josh me about my height and I just point out that I rarely bump my head and I fit comfortably in sports cars!
Even dating, you need to find a single partner. Maybe two if one passes before you or you suffer a divorce. Given modern dating technologies, it’s never been easier to find a partner who is almost certainly already aware of your various physical attributes prior to even meeting. OP already had a partner though, so this isn’t even a concern.