A dirty fuel filter could do this, but like I said your car doesn't have a normal fuel filter. It only has the pump inlet screen which is kinda like a mesh bag around the pump, inside the gas tank. You'd have to get a lot of mud in the tank before that's a problem. Still, you can keep it in the back of your mind as everything else gets ruled out.

I'm inclined to believe the dirty idle valve, since driving around is OK. If cleaning doesn't do it, those valves sometimes just go bad.