One of my aunts said something to me about early retirement that lessened my Protestant Work Ethic guilt, which was "when you retire you will be allowing someone else to take your job, who may actually want it".
My parents said this same thing. I think a lot of the guilt people feel over ER is either 1) Protestant Work Ethic, or 2) seeing people you care about being stuck on the treadmill.
The Protestant Work Ethic is something that our society has forced into the "paying job" mode, but it doesn't need to be satisfied only with a career/job. If you're feeling guilt because of this, sit down and have a little chat with your guilt. You may be able to fulfill the Work Ethic with other good works. This doesn't mean that you should fill your retirement with slaving away for others, just that there may be something you can do that genuinely meets a need to feel useful. Your job doesn't/didn't necessarily do this!
Seeing people you care about stuck on the treadmill is a more complicated thing. If they're genuinely hard-working, but haven't had the opportunities you've had, are there ways you can help them? If they've chosen to spend at or above their means, you may need to remind yourself that they're adults who made those choices due to their own cost/benefit analyses, and resolve to be there for them to give advice if they ask for it in the future.
If anyone is trying to guilt-trip you, however, tell them to take a long walk off a short pier.
Lastly, all of us who've been able to save a lot of money have inherited lucky genes, in any number of ways (and have had other random good luck in life events). There's nothing wrong with acknowledging this and it's nothing to be ashamed of, it just is. It also exists alongside the hard work you've put in, and doesn't lessen the pride you should take in your hard work.