I was thinking more along the lines of Idle Air Control motor. Sometimes if they get carboned up, you can get some really goofy idle conditions. It could be EGR as well. Have you tried running some fuel injector cleaner through it?

Also, is your check engine light coming on during any of this?