I do it a lot and haven't seen any damage from it but if done from too high an RPM range then it can cause some severe problems.

In otherwords don't downshift from 4th at 60 mph to 2nd because your trannyu and engine won't like it for too long.