I find myself doing this sometimes too... once I actually notice it (usually after I've done it a few times in one conversation), I'll try to cut it out for a while. I definitely see how it could be annoying on the other end.