If you are worried about it getting too large and never being able to use it all for medical expenses, start pulling from it with your current and saved expenses now. This will slow the growth compounding.
IMO the largest benefit of an HSA is the reduction to taxable income, which is a benefit at your marginal tax rate.
Tax free growth is a great benefit too, but if you let investments in taxable grow, you get favorable LTCG tax rates, so the tax free growth is only a benefit at your LTCG rate, which is going to be smaller than your marginal rate.
From these two factors it is more beneficial to max out your HSA contribution advantage than to keep tax free growth going, if you have to cut back on one of them.
Tax free withdrawals are another benefit, and you get that whenever you withdraw from the HSA with medical expenses. If you let the HSA grow more than you'll ever use, the overage is essentially like a tIRA, but worse to leave for heirs since they must withdraw it all in the year you die. So, you are correct to be looking at letting your HSA grow too large.
The catch is, you really don't know your future medical expenses. First, you don't know how long you'll live. If you live long enough, you know that you can use your HSA on part of your Medicare premiums, but you have no idea how much you'll pay on actual medical expenses.
