Put the cat back on. It is probably hurting your low end torque badly without it and it quiets it down a great amount and keeps the air clean too. :thumbup:
Backpressure is a myth. It is flow velocity that you have to worry about. And it doesn't increase compression at all. Since it sounds like you have a stock engine, keep the exhaust on there, put a cat back on, and keep the header. If it isn't quiet enough, put a quieter exhaust on.
If you are so anti-noise, it is worth the 3-5 hp you gain from bolt ons to keep them?