Its possible that the RCA marked "Sub" will be a mono output (as you don't need left and right channel for pure bass tones). The other RCA outputs will be for normal speakers inside the car (should you decide to run them off a seperate amp).
With the stereo turned off, check that the amp is getting 12v on a volt meter. When you turn the stereo on, check it again... it should remain at 12v... if it drops to 10v, its possibly and earth leakage issue and you'll need to check you cabling to make sure everythings secure and you have a good earth.
hope this helps