Broadly you're of course correct. Few of us amount to anything, and most aspirants will be woeful failures. But it takes tremendous self-awareness to let go of this "conceit", especially for those of us who did actually publish a thing or two, influence such-and-such, and got some smattering of publicity. Utter failure would have almost been easier to bear, than partial failure.
Let me assay with an alternative vignette. Bobbly is a bright teenager, much talented in math. His parents want for him to go into some STEM field, maybe do research, maybe start his own company and make lots of money. But instead he bums around, barely makes it through school, skips college and goes into some quotidian trade or whatnot, earning a living, but not much beyond that. What do we say about that? Was Bobby's choice a good one? Do we have no standing to judge, because it's only his life and his alone?
Alright. Let's shift the goal posts. Alternative-Bobby gets fantastic grades in high school, and goes to Harvard on a scholarship to study math. Becoming a superstar, the professors in his department are falling all over themselves, to lure Bobby to become their graduate student. He spurns them all, instead leaving to become an insurance salesman. Now what do we say? Yeah or nay?
Continuing, alternative #2 Bobby gets a PhD in math, and goes on to get an assistant professorship at some university. Rising through the ranks, he proves an important theorem... enough to get the Fields Metal some years later. But then, seemingly at the height of his career, he resigns his professorship, at age 40, to go become a gentleman farmer in New Zealand. Now what do we say?
Alternative #3: this Bobby remains a practicing mathematician into his late 50s or early 60s. Feeling no longer sharp, he leaves while still enjoying maximal dignity, prestige and fame, quietly retires.
And, alternative #4: in this version, Bobby never retires. He doesn't even go Emeritus. In his 70s, he shuffles from his office to the coffee pot, barely coherent in his lectures, abandoned by students, unable to garner funding, unproductive and widely lampooned behind his back. He finally dies of a heart attack in his office.
Few of us would prefer Alternative #4 - though in truth, I've seen it more often than not. #3 sounds more reasonable, or perhaps, depending on lifestyle choices, #2. But my point here isn't to argue for or against early retirement... after all, this Forum has a certain thematic leaning. Instead I call attention to the original Bobby, the first alternative, and the second. Wouldn't we say, with sound basis, that the first version of Bobby failed to rise to his potential? He mishandled his manifest gifts, and the world is the poorer for it, is it not? Well then, at what point along the chain of alternative Bobbies, do we decide that OK, our hero has done enough, and is honorably ready to move on?