Like most of the others, I think the single stock risk is the overriding factor and in your shoes I would diversify away from it at some pace you feel comfortable with.
The other tax point that I wanted to make and that your tax person didn't consider is that starting SS now and probably continuing some capital gains would mean that more of your SS would become taxable. Take a look at the worksheet behind Form 1040 line 6b, or ask your tax guy about it and he should be able to explain the basic idea and do some example math for you.