No, not at all. You do the test with the engine off. The tester plugs up the turbo inlet. The air is stopped by the valves, so the test includes your engine's throttle body, intake manifold, and all of your vacuum lines. I've had professional mechanics tell me I was crazy and that this would never work because the air will leak past the valves. WTF? At no point does your engine have all 4 valves in any cylinder open at once. DSMers have been doing this for 14 years. It works. If you have a bent or burnt valve, it won't, but it will help you diagnose a healthy motor.

It will and should leak very slowly past the rings unless you oil all of the cylinders first, but by no means do you need to do that to test it. It's easy enough to just blow some air pressure into it, listen for leaks, and if you're not sure exactly where it's coming from, spray it down with soapy water and look for bubbles. You'll definitely see them if it's leaking. You'll be surprised at what you find. I've leaked boost from my TPS sensor harness (electrical part) before... where the actual wires were coming out. Bad throttle body shaft seals were the cause.

Putting a gauge on your tester (I used a $2 tire gauge and some $.50 NPT fittings) lets you know at what point you'll blow your IC pipes off (not so funny story), or at what PSI your BOV/etc. starts leaking. You can get a friend to watch your gauge, but the one on the tester is the shit because you can do it by yourself and get better info.