sounds like your brake master cylinder is faulty get it replaced asap as it will get worse
one sign is if you look under the master cylinder you will see fluid leaking on the booster but not always .