A large diameter muffler ("fart in a can exhaust") will not "kill" HP because the muffler does not create any backpressure. It's like sawing off the last 2-3 feet of your exhaust pipe, IF anything it will decrease (very slightly) the amount of backpressure the engine "sees", increasing the amount of high-rpm horsepower. But, basically any muffler put on a stock diameter exhaust pipe system will do nothing for performance...
Torque is another matter altogether, and usually the less backpressure the less torque is available down low in the rpm range, so obviously it's not good to have a 4" Deisel truck size exhaust on a 1.6L NA Honda motor...
However, are you talking strictly about the muffler or the entire exhaust system from the cat back?

Yes, a larger pipe can be beneficial, ONLY if it is used throughout the exhaust system...

So, you "heard" wrong, make sure you correct the guy who told you that (my guess is someone who doesn't like civics lol)
As for drifting, I don't know about that (I do not follow it much), but I know for a fact that Forced Induction vehicles (alot of drifting nissans are turbo) will benefit greatly from a larger than 2.5 in exhaust pipe. just go ask any DSM owner, they'll tell you 3" all the way!!