Does it actually get to the floor on those occasions or is the pedal just real spongy and the more you push the closer it gets to the floor?

If it's the first thing than the booster sounds like the best guess to me, if it's the latter, than doing a full brake flush might help. Whether it's air in the system or that the fluid is just too old and full of water, etc. like mine was.