Thanks for all the responses. We decided we are going.
I rationalized that it would be like avoiding Manhattan because downtown Albany burned down. Doesn't make much sense and as many have said, Hawaii's economy depends on tourism so staying away from non-impacted areas would hurt more than help.
FWIW - I agree with your decision. I'll add a datapoint from personal experience.
Hurricane Irma hit the lower Florida Keys in early September, 2017, IIRC, as a Cat 3. As you can imagine, widespread damage. For some period of time, only residents were allowed in.
We had a condo booked on Stock Island (last island before Key West) for a month starting in mid October, I forget exactly when. Booked it directly with the owner through VRBO. There was no significant damage to their complex, occupancy was no problem, but the owner was willing to refund the cost if I chose not to go. We debated, but in the end decided to go.
Turns out our start date was about 1-2 weeks after they started letting non residents in, so we were among the first tourists to return. Drove through some heavily damaged areas, and there were debris collection sites along the side of the highway. Constant stream of semis being filled up with a large loader, probably took a few minutes each, as soon as one was done they'd go to the next. It was tough to see the destruction. As a teenager I also volunteered for a few days with the Red Cross following a significant tornado outbreak in my region, which required accessing restricted areas, so it was not the first time I witnessed the aftermath of powerful storms.
Plenty of restaurants were not open yet, or opened during our stay. We ate in the condo a fair amount, but when we did venture out things weren't very busy. Every single person we came into contact with at stores and restaurants was glad we were there. We told them we had debated making the trip, and they would discuss what they went through, there was a wide range from minimal issues to homelessness. One waitress said her rental house was OK, but many friends were displaced, they had something like 12-15 people sharing one bathroom and the kitchen, some camping in the back yard and others crashing on the living room floor. Just helping each other out. I would typically tip around 50% and up, considering it my contribution to their recovery, and got more than one hug from a grateful server.
As has been pointed out, these were tough times for the residents/workers, but they needed the income more than ever.
As for our vacation, it turned out to be great. The complex we stayed in had two other guests while we were there, one was a couple for a week and the other a nonresident owner who came south for a weekend to check on his place. Had the pool to ourselves most of the time. The main disappointment was that I had planned to do some snorkeling, but the water visibility was very low from things getting churned up by the storm.
Enjoy your trip!
