We lived in Boulder for the better part of 20 years, Cañon City for a couple of years, Silver City NM for a couple of years and have traveled extensively all over both states.
I agree that the entire Colorado Front Range is just nuts now. We still have friends there and my sister lives in Denver but between the traffic, prices and wildfire smoke we avoid the place except for flying in for a short visit occasionally. A very outdoor-oriented cousin lives near Cortez and loves it but it's very small and rural. Hard for me to see the appeal of Grand Junction but YMMV. Cañon City has the mildest climate in the state, great cycling (mountain and road) and nearby hiking and is only 45 minutes from Colorado Springs shopping but is ultra-conservative and culture less. Manitou Springs, while touristy, is the Boulder of Colorado Springs and quite livable but as cold in the winter as anyplace else in the Front Range and far from cheap.
Regarding NM, a lot of people don't know that Albuquerque has one of the widest ranges of altitude and microclimate within its city limits of any city. The NE heights for example are more like Santa Fe in terms of altitude and weather, with great access to hiking in the Sandias.
We spend a few months in the summer in Santa Fe quite often and while the hiking trails are great the restaurant scene is far inferior to ABQ unless you mostly like to eat at high-end places and don't mind paying silly prices. ABQ has great pizza, bread bakeries and coffee as well as good Asian cuisine, while Santa Fe has none of these things. With over half of Santa Fe's workforce commuting in from ABQ everyday it has become a "toy town" for tourists and Trustafarians - much like Boulder and Colorado's ski towns. ABQ's crime levels are off-the-charts awful but I'd still choose it over Santa Fe.
Las Cruces may be the second largest city in the state but there are more things of cultural, culinary and artistic interest happening in Santa Fe or ABQ in a week than there are in a year in Cruces. The summers are as scorching as Tucson where we live now, the hiking is mediocre at best, and as for food, hope you like green and red chile 'cause that's pretty much it. And having El Paso as your most sophisticated big city shopping option (and the nearest place with a Costco) isn't exactly thrilling. It's NM-meets-Texas.
Silver City is lovely but very, very small: less than 10K population. The hiking and biking is the best I've ever experienced - far superior to anyplace in Colorado IMHO - but more than half of those who move there for retirement end up leaving. 3.5 hours to Tucson for your nearest Costco or Trader Joe's (or good non NM cuisine restaurant - or an airport) gets old in a hurry. Good place to be a hermit though, or if access to pristine trails trumps every other priority.
In any case ABQ is the only city in NM with complete health care services. Santa Fe friends have to go there to see specialists and Silver City folks are often flown there in case of emergencies as there are few specialists at its hospital. Meanwhile Las Cruces friends who worked in medicine for decades go to the Mayo Clinic in Phoenix for anything serious. NM is indeed beautiful and captivating but it's also extremely poor, exceptionally crime-ridden and has some of the lowest educational levels of any state, along with far and away the highest levels of alcoholism and drunk driving. It's also incredibly insular, culturally - something that many who move there from elsewhere only discover after several years of frustration.