Sounds like the starter is on its way out. If the problem gets worse, I would replace the starter.

It may be having issues keeping the gear engaged or disengaging it, so it grinds.