I think you are right. I was recalling the last time I rolled a 401k, which was in 2001. At that time, you needed to keep a separate Rollover IRA account if you ever wanted to roll the proceeds to a new employer's 401k. If you mixed it with your other IRA assets, you would be precluded from doing so. However, it appears that the law was changed in 2002. Now, it may only be a problem with the new employer's plan trustee rules, not an IRS prohibition.
In addition to the 72t issue I mentioned earlier, it would also be helpful to keep your rollover amounts (which are 100% taxable when drawn) separate if your regular IRA has any non-deductible contributions (which results in less than 100% taxation depending on the ratio of non-deductible contributions to the total amount)
My apologies to Wormrider for the earlier misinformation.