However, the culture at our agency was not as you describe.
I didn't describe the culture at your agency.
As I said, not all fed jobs are the same. I'd guess that the AUSAs in the SDNY office have some pretty intense jobs. And I'm pretty sure they routinely work more than 40 hours per week without having to get authorization.
But, in my experience, most lawyer jobs at agencies do not have the same culture as major law firms in terms of how demanding they are. The fact that the hours tend to be longer and the expectations tend to be higher for partners at law firms does not mean that the government lawyers are "slackers" as you put it or "untalented." (Your terms; definitely not mine.) It does mean that the hours, demands, and cultures are different.
You emphasize that your job was a 40 hour per week job and people took their vacation. I would bet a lot of money that the OP did not become a partner working 40 hours a week and taking significant amounts of vacation, and I would bet even more money that they are not the only attorney there on weekends. I also would bet that their standards are beyond demanding good or very good work.
So, what happens when the OP gets work at the new job, feels like it's necessary to spend more than 40 hours a week to do all the work and do it to their usual standard? As a new federal employee, they wouldn't have much vacation time to give up. But, they could give up evenings and weekends. And for much lower pay. And what if a pattern develops and supervisors like giving the OP more work and more demanding work knowing that they can count on the OP to get it done? The OP says that they do not find ER appealing and that they ended up as a partner at a major firm out of "inertia." That suggests that they inherently are a person who demands a lot of themselves. That's unlikely to change when they leave the firm.
Also, just because the OP files petitions with the agency does not mean that the OP knows what it's like to work there. Those are two very different things. If people are leaving the fed jobs there as the OP says, then I definitely would want to know why. I don't get the impression that the OP necessarily has a great understanding of what it would be like to work for the government. If the OP doesn't want to get an in-house job because of internal politics, then they definitely need to talk to more people about federal employment because lots of federal jobs come with lots of unpleasant and frustrating politics.
I totally get the desire to flee a law firm. Been there. Done that. And much more quickly than the OP did. But, I also understand how unhappy a person may be when they flee the firm for the federal job.
But, everyone is different. I don't like my current job for a variety of reasons. My friend doesn't understand why I don't just quit. When I was younger, that's probably what I would have done. I realize now that there's not necessarily a job I could get that I would like more. And my sense is that I could retire in a few more years if I could stick it out with this job that long. If I left for another job, it might be ten more years before I could retire, and I might be just as unhappy with that job.
Bottom line: I think the OP should make sure that they really understand what they would be getting into and be very honest with themselves about their own characteristics and whether making the change is a good idea.