Eh, I sort of think of it along the lines of "When you tell somebody somethin', it depends on what part of the country you're standin' in... as to just how dumb you are." (courtesy of Burt Reynolds)
Meaning, that words and phrases have different meanings to different people. For instance, sometimes I'll just say "I can't afford it" and leave it at that. "I can't afford it" is just a broad phrase, that can mean a broad variety of things, to "buying/doing that would bankrupt me" or "it's really not in the budget right now so I'd have to plan for it" to simply "I don't feel like paying for/doing that, even though I easily could" or "I don't see the value of wasting my money on that."
Plus, usually "i can't afford it" tends to stop the discussion cold, in my case at least.
This friend sounds like they could be one of those types who has to comment on everything, and constantly judge/value things. Upon hearing "not really in the budget right now" most people would take the cue, and know to drop it and that there's nothing to debate. But no, this person had to close with the parting shot, the judgement call of "maybe you retired too early."
Now, if this person was a good friend, where you'd regularly talk about finances, retirement, budgeting and so on, and someone with whom you shared that type of camaraderie, I could see that type of comment being brought up. But, considering the circumstances of the meeting (work friends at a party out of state), and the fact that the comment stung, I gather this is not that type of friend.