when you release that throttle, your engine is getting little to no air into the cylinders, but you ECU is still sending a little fuel into the cylinders. there isnt enough air in the cylinders to ignite the fuel, so the unburnt fuel enters into the extremely hot exhaust manifold, which does have enough air for ignition. so small amount of fuel ignites in the exhaust creating a popping noise. you cant hear it with the stock muffler on because it does just that, it muffles the sound to the point that you cant hear it.
in other words, unless it starts getting very loud and you are shooting fireballs, dont worry about it.