Although... many of us have takend it off due to anoying rattle and have had no problems.

That doesn't mean nothing will happen later or if the reprecussions aren't simply noticable for the time being.

IMHO, take it off baby!