It probably just causes a little extra wear on the throw out bearing. I almost always leave it in neutral too.