Keeping them open unused doesn't hurt, but if you want to downsize your collection, I'd do it gradually, and usually starting with the youngest account first (unless you use it most because it gives the best rewards). In general, I'd focus on newer accounts, accounts with lower limits and cards that give no rewards (or the least rewards). I'd keep older accounts, accounts with higher limits and those with the best rewards programs.
In reality if you pay off each month and use 10% or less of your available credit each month, you're probably OK to just trim half of your credit limit and still have an excellent credit utilization ratio. That said, age of accounts matters so I'd be hesitant to close an account that was, say, 20+ years old.