Australia may (probably will) tax you. The U.S. will also, as all U.S. citizen income is taxed regardless of where it is earned in the world. There may be (probably is) a tax treaty between the US and Australia that will allow you to take credit for any taxes paid to Australia, but you must still report the income on your 1040.
And don't forget that you will also pay taxes on any currency gain (and get a deduction for losses). Thus, for example, if the USD and AUD are trading at parity and you buy a $1000 1 year 5% CD denominated in AUD, and over the next year, the AUD rises to $1.05 USD, you will report $100 income when the cd matures -- $50 of interest and $50 of currency gain. That currency gain is ordinary income, unless you meet some tricky rules to be designated a currency trader. Look at the IRS website.
Buying AUD denominated cd's through Everbank does not expose you to taxation by Australia