Serpentine belt? So it's probably got a spring-loaded tensioner & at least one simple idler pulley. Look for noisy bearings in any of those. Poor-man's stethoscope is a short peice of garden hose (18" or so) held up to your ear...

Like you said, running without the belt won't help differentiate. And if that belt runs the water pump, you really don't want to run without it.