I was pulled over once for my exhaust back when I just had I/H/E because of a noise violation. I told him that the exhaust was 50 state legal because it was under 95 dB at WOT, so he looked at his code book and gave me the benefit of the doubt.

In your case, I would keep that silencer in. If you do happen to get a fix-it ticket, there is no guarantee that it will pass the noise standards in your state.