In a sense it doesn't matter. But, I was actually thinking about this the other day. Class has a number of factors to it. Income is part of it, but so is education. Some activities are more tied to certain "classes" than others. Some of the people I know who say class doesn't matter are people who were raised in one class and stayed there. It can be different if you changed.
I was raised in the working class. I lived in a neighborhood where most people had good jobs that didn't require a college degree. When I went away to law school I was suddenly around people who grew up in a very different environment. And, yes, there was a gulf at times between what I knew and thought and what those around me knew and thought. There was a difference between middle class (what I had considered myself growing up) and upper middle class (what I felt everyone around me was when I was working).
When DH and I were working full time we had upper middle class incomes and lived in that type of neighborhood. Maybe because of my upbringing we always did some things that were more middle classy. I liked chain restaurants that some of the people I knew barely even knew existed.
But, now with DH retired and me working only a few hours a month it is different. Our income is definitely a middle class income. We have more assets, of course, than most people earning that income for a living.
We are planning to move soon and I have thought about this when moving. We want to downsize and want a less expensive home. We currently live in one of those upper middle class gated enclaves. So most of the people in the neighborhood probably have college degrees. A few may not who have incomes though that are higher than average.
But, I was looking at some houses where we plan to move. The area is very safe. That isn't an issue. It is close to amenities. Within a very short distance is a very high income area that we don't want to buy in because we don't want to spend that much money on a house. Back when we were working we would have done that.
So, this area I am looking at has some smaller houses (which we want) and is safe, etc. But, I looked at the demographics and it has a very middle income average income. Only 1% or so of the people have graduate degrees. It is very clearly a middle class area. And, I did wonder what it would be like. Would I fit in with the neighbors? I mean in terms of interests and stuff like that. It was very clear years ago that I didn't fit in that well with the part of my family that didn't go to college and stayed in the same/similar areas that I grew up in. I still loved them as family, but we had little to talk about once we got past family stuff. So, I do worry a bit about moving into an area where we may not have that much in common with neighbors.
That said, I don't want to pay to live in the kind of area I would have lived in 15 years ago.