Garageband has a lot of power, but Apple also sells Logic, which is their serious production software. Garageband is a nice, free introduction to see what is possible, but it's not used nearly as much as Ableton, etc. - and it's kind of a closed system of software and add-on packs. What are you missing in Ableton that makes you want to change? Garageband is excellent for beginners, but you sound like you know more than that and may run into some limitations.
There are many excellent Mac DAWs/production tools and it's worth your time to do some research on the best software for you. If you want something totally free and very powerful to get started, go with Reaper. It's the most popular shareware DAW, and you can use it as long as you want without licensing it. It also runs plugins and other external software instruments, many of which you can also get for free. The nice thing about Reaper is that it is very easy to learn if you've worked with any other DAW. There's nothing really quirky or complicated about it. FWIW, If you Google "Top 10 Mac DAWs" you'll get the list of most worthy contenders for real music production. They're all good.
I've worked on Digital Performer for decades, but I've tried then all and I own a copy of Live. Live is really good, and there's another competitor called Bitwig that's very cool too. Funnily enough, I've very recently made the choice to switch from DP since I have to buy another upgrade to go forward. I have a UA interface, so I'm going to try their new Luna DAW (free for owners). Their plug-ins are so good, but their DAW is very new. If I don't like it, I'll upgrade and go back to DP.
