It's definitely a vacuum problem, but it may not be in a vacuum line.
Check the seal around the brake booster (Round thing on drivers side of firewall) to see if it is leaking...
A buddy of mine had same problem, we couldn't figure it out. We knew it was a vacuum leak, but we checked and re-checked every vacuum line. Turns out it was the seal around the brake booster...
Check it out!