Yep practice usually works wonders.

4 years though? hmmm...maybe it is your practice technique. I learned to do it through repetition of simple beats and slowly speeding up the count. As soon as I missed a beat I would slow it down again and keep it there for a few minutes then speed it up again.

Start real simple and as you get better add another movement to it. It's a bitch once you start doing a bass and a hi hat along with hand motion around the kit..gets confusing fast but if you know the music it gets easier.