This is relative to society's expectations. Running with the theme of "graduation", consider the age of graduation for high school. Normal age is 18. A person who does it at 17, might be lauded as being quick or talented. It's something of which to be proud, or in any case, not to hide, even if it's awkward to still be only 17 during the following autumn, as a freshman in college. But what if a person manages to graduate from high school at age 14? That would be freakish and bizarre. How would a 14-year-old mingle with peers, advance to the next stage (presumably college) and so on? Some cover-story would be needed, or at very least, very ginger and deft management of one's reputation.
So too, I think, with early retirement. Early-ish retirement, say at age 60, is no problem. Some people might be envious, but to retire at 60 is something of which to be proud; no explanations or excuses are necessary. But at age 51? That's akin to graduating from high school at age 14... even if, at 51, one might plausibly have had a 30-year career. If indeed retiring at 51, one likely has to spend some years - maybe 5 or 6 - with a low profile and a cover story.