A few years ago they told me that I need to get them taken out (all 4).

I just never did, and it's never caused me any pain or anything. Teeth aren't fucked up at all either.