It does not work that way.
First you report income. For capital gains you will report this on 1040 line 14. This whole section is INCOME. Most of it is taxed, some of it is not. The tax rates vary.
Second you have adjustments to income. Here you could deduct student loan interest (1040 line 33) or take a tuition and fees deduction (which is really an adjustment- 1040 line 34).
An adjustment is subtracted from your income (income-adjustments=AGI).
On page 2 of the 1040 you have taxes and credits as the next section. Biggest difference here is many of the credits are AGI driven (where as the adjustments before might lower AGI for other credits to come into effect).
1040 line 43 is TAXABLE INCOME. This takes into account your deductions (standard or itemized) and your personal exemption(s).
Taxable income is located on tax tables, combined with filing status to determine your TAX. 1040 line 44.
Then you look at the credits you may qualify for. A credit reduces the tax you owe dollar for dollar. So if the hope credit of $1650 (max credit) is what you qualify for, then you would subtract $1650 from the TAX calculated on line 44.
If your tax is ZERO, the hope credit is non refundable- meaning it cannot reduce your tax liability below zero.
A few points- the lifetime learning credit could get you back $2000 instead of the $1650 for hope- is there a reason you are not using the lifetime learning credit? Also consider taking the tuition and fees adjustment. You can only take ONE of the three for a givern person in a given tax year. Take the one which is most beneficial that year.
Only time I would take the hope is if
a) tuition was not very high (less than $2200)
b) there was another person on same return using the lifetime learning credit
c) there was another situation which made hope more advantagous.
FYI lifetime learning gets you back 20% of tuition up to 10k (so max credit of 2k) if you qualify.
If the capital gains triggered an increase in tax liability, then yes the credit would offset the tax incurred. If the capital gains was already included in the situation where your tax liability was zero, then the credit is meaningless to you for that tax year.
Light travels faster than sound. That is why some people appear bright until you hear them speak. One person's stupidity is another person's job security.