The ball has to be able to spin in it's socket when you steer, right? It's only bad if it can rattle around. Or if the entire socket spins in it's mounting.

I've had to use a jack or crowbar or something to press the taper shaft tightly into the tapered hole so it doesn't spin. That's not too unusual.