The only way it wont be off is if it is at the factory ride height. Whether or not you want to fix it is up to you. You'll be looking at uneven tire wear and possibly some handling issues, depending on how bad it is.

As an aside: my camber is off even at the factory height.