Check the vacuum line going to your brake booster, and inspect it and the booster for cracks. You engine looses those rpm's at idle because you're creating negative vacuum... or that is to say... you're making the vacuum that the engine creates work harder, in order to actuate the brakes.

For a lot of fun, take that line off the booster and plug it. I hooked my intake manifold up wrong, and left that line off once. You have to almost stand on the brakes just to come to a semi controlled stop.