the way you explained it, it could be vaccume leaks. Other problems I've come by was the Idle Air Control Valve gummed up and I had to blast it with carb cleaner. The fast fluctuation of the RPM also could indicate that the Throttle Position Sensor is out of wack or just unplugged. But a vaccume leak of the intake manifold fits better since your brakes escalate the problem.

hope this may help