Well in my case it turned out to be the fuel pressure regulator.
I recently installed a b&m to lower the idle fuel pressure. It must have been faulty or something because as soon as I put the stock one back on, the problem went away.