The timeshare industry is well-known for high-pressure sales tactics often to people who are unknowing of how time shares work, often can't really afford them and are unaware of what a financial drag that they often become. Sponsors often make dubious claims as part of their high-pressure tactics. It's foolish to think that these timeshare sales are on the up-and-up.
Look at the OP... paying $4,128 in annual fees for a timeshare that originally cost $20k. OP didn't say how many nights he has but even if he has 3 weeks that's ~$200/night and if he has 2 weeks that's $300/night.
We've all seen the stories, you really aren't presenting anything which is unknown. The world we live in is buyer beware. A salesman is not there to determine the suitability of any product or service for you. He/she is there to sell you something. By you going to the seller, you are obviously there as a potential buyer. If you are unable to control yourself and understand your own finances, what you can or cannot afford, why is that anyone's responsibility other than your own? Again, who goes to a timeshare presentation not knowing what they are there for? As far as I remember, every one of them makes you sign a disclosure form that you understand that your are there for the sale of timeshares, and that you represent you have a job and/or income above a certain amount. And once again, there is the week or so that you get where you can unilaterally back out and cancel the entire purchase.
The OP has not been fully clear - I seriously doubt that annual fees are $4128 in his case, he likely financed his purchase to some extent, likely at an interest rate well above market rates, and that annual amount is likely the all-in amount with his financing and annual fees.
When DW and I were first married and we were visiting Disney, we went to a timeshare presentation because they were giving free park tickets if you sat through the 90 minute presentation. It was a hard sell. But we pleasantly smiled, played devil's advocate, and at some point it was clear to our assigned guide we just wanted the park tickets. After the tour, when we sat down to talk turkey, he put the papers in front of us which indicated financing at 17% and I couldn't stop laughing. All the while they were ringing some stupid bell every time there was another (supposed) sale.
For years the experience had us so turned off on timeshares. And then on one visit back, I woke up early one morning and went to the Disney sales office - as it was just a short walk from where we were. It was like another world. The meeting lasted maybe 20 minutes. The guide said no pressure, here's what we're offering, here's the prices, if you like it wonderful, if not, not a problem. Maybe a month later I called back, gave the guide our AMEX number and put the entire purchase on the card. That was 20 years ago, and DW and I always wished that we had gone to Disney a few years earlier instead of that other one (now owned by Sheraton/Marriott) which turned us off. The sales guide who sold us the membership 20 years ago is still there, when we visit he routinely leaves us a voicemail. Nice guy. As I mentioned on an earlier post - we can sell the Disney timeshare today for double what we paid. However, Disney has delivered everything they promised and more. No way we would give it up. It's paid for itself many times over. So again, not all timeshares are the same or necessarily bad.