Hopefully you didn't knock out the outer shell of the OEM one and that's why the ES ones went in by hand. I had to use some circular plumbing parts and bolts to push mine in.

I've read about using a hammer to knock the old ones out and new ones in. I will be attempting this method since I now have 5 cars to do. I'm currently using the ES ones on my current driver, but they're starting to wear and I would like to get back to the rubber ones. The other 4 are just ripped OEM ones.