If it was the brake rotors it would also vibrate (and worse) when you apply the brakes.

That does sound to me like a wheel or tire causing the vibration (my first guess), but it could be something else. I'm not a mechanic.

Does the vibration change or even go away if you lift completely off the throttle at those speeds?