It could be your ignition timing. Get someone to put a timing light on it and see where it is. What your engine is doing when you're coming off-throttle is normal. When you apply brakes, you're pulling vacuum from the engine for the power brake assist diaphragm. The engine senses that as being under load, and it will adjust it's idle until the car stops or until it's no longer under load.

Might want to check all the vacuum hoses on your intake manifold and make sure none of them are dry rotted, broken, or loose. Vacuum leaks cause idle problems, too.