check you vacuum system... My buddy had a smiliar problem, he had a slight vacuum leak around the brake booster (the big circular thing on the driver side of the firewall), although his car was built from a junkyard rolling chassis...

However, did you do the swap yourself? Or if someone else did it, maybe they "bumped" the brake booster and it lost it's vacuum seal...

Check it out...