Part of your exhaust is rattling. Do a search for exhaust rattle. Something's rubbing or one of your exhaust hangers is broken. The reason you don't hear it when you downshift is because the engine torques in the opposite direction.