I'm not sure I understand what you guys are thinking but that's no reason to hesitate to muddy the waters more

:
Start w/ 500K in a pretax rollover IRA from a corporate plan. Add a 5K
non-deductible contributory TIRA. Because the contributory TIRA is non-deductible, you have to pay taxes on that income =25% of 5k = $1250.
Later (make it a yr later just to separate the concepts), you decide to convert 5K to a Roth. Even tho you "physically" convert the contributory TIRA, the IRS doesn't care that you already paid taxes on those $$$$. It considers both assets as one big uniformly mixed IRA (like cream in coffee).
5K of the 505K is basis .....previously taxed(1/101 of the mix; slightly less than 1%).
When you convert 5K to Roth, 1/101 of that or slightly less than 50 is considered basis and is not taxed. Slightly more than $4950 is taxed at
25% or about $1238 is taxed. You end up with 5K in Roth and 500K in
TIRA with about 4950 in basis that won't be taxed upon withdrawal.
It may seem that you are "double" taxed if you insist on thinking about converting the "non-deductible " TIRA but that is the wrong way (but perhaps natural way) to think about it. Think about the cream in coffee concept and you will see that it all ends up correct in the end.
Bottom line.....I'm more agreeing w/ soupcxan.
btw.....there is a way to make things better. If you had a current employer plan that would accept your previous employer's plan, you could move the 500K rollover IRA there, leaving the 5K non-deductible IRA.
Then when you converted that 5K to Roth, no taxes since it is all basis
(perhaps some small earnings component would be taxable).