It's pretty much impossible to do carry-on only with scuba gear. I don't mind checking bags but if you are opposed to that you could also consider shipping like UPS, DHL, or even the airlines are offering baggage delivery. Obviously you have to offset that cost with the cost of rentals and the thoughts of cooties. The saying is, there are 2 kinds of divers - those who have pee-ed in their wetsuit and those who will. Most with over 10 dives are in the former category because it is just not a big deal.
If you are willing to kick a little harder and get full foot fins you can save a little on space by skipping booties and using shorter full-foot fins. I have always used full foot fins but as a divemaster the shop required me to use booties and strap fins because...more pieces of gear = more sales. So I have that setup but don't use it for personal diving. But my full foot fins are longer and don't fit in a carry-on.
If we were just going to dive, I would have no problem checking our gear. The issue is that we’ll be flying from Sydney to Tasmania to Melbourne to Kangaroo Island to Cairns to Brisbane. That’s a lot of flying schlepping SCUBA gear that will only be used in one of the destinations. Prefer to avoid that.