After thinking a bit more about this, I think the rate you ultimately will come to rely on depends on how much flexibility you have and how long you'll need to rely solely on your retirement funds (social security or paying off mortgages). I'm okay with 4% because we have a fair amount of flexibility.
Flexibility is having some moderate luxuries in our spending (like some travel or other non-essential spending). If the market tanks and and I see the danger of sequence of return risks, we can pretty easily lower our spending for a year or a few maybe 3.5%.
We have a rental so while at the beginning of ER, we are drawing 4% to cover our spending, over time rents will increase and eventually the mortgage will be paid off so over time, the cash flow from the rental will increase substantially and the withdrawal rate should go down. I'm projecting that by the time I'm 80, the net rental income could entirely cover our spending (however, it'll cover an increasing amount before then).