For me at least, this seems like one of those situations where there really isn't a right or wrong answer. With $5.5M in invested assets and only a $500K mortgage, to me it seems like you've already won the game, and whether to keep or pay off the mortgage, it doesn't really matter either way.
I'm guessing the principal+interest on that mortgage is about $4,000 per month? That part would bug me, but I'm also looking at it through the eyes of someone with a principal+interest payment of about half that. But also, about half as much in invested assets, so I guess it's about the same, proportionally. For me though, it's a 30 year mortgage at 2.875%, so I've chosen to just let it run its course.
In my mind at least, while 6% is a little high, it's sort of in those in between stages. If it was a 30 year mortgage, I might be tempted to pay it down more quickly, but at a 15, it's already paying itself down pretty fast.
My first mortgage, taken out in December 1994, was 9.625%. Now that one, I tried to pay extra on. But, in 1999 I refinanced it to 7.25%, I believe. At that point, I didn't feel a pressing need to pay any extra on it, but at the time the stock market was also pretty hot, so I thought I could make more money investing. But, fast forward to early 2003, and I refinanced again, this time to a 15 year at 5.5%. I figured doing this refinance was sort of a combination of getting the benefit of a lower rate, and forcing me to pay it down faster, and I was happy with that at the time. I ended up selling the place a year and a half after I refinanced.
If it was me, I might just keep the 15 year mortgage, as is. But where I might go against the grain, is if mortgage rates drop enough, refinance it. And if the interest rate between a 15 and 30 year mortgage isn't much different at the time, might even go with a 30. But, I'm really not concerned about being mortgage free. To others, that's very important, so I can see both sides of it.