You pretty much have to remove all the trim in the rear to remove the rear deck, which has to be removed to access the speakers.
The top side panels come all the way out, the lower side panels can just be popped a little and pulled out of the way. There are three clips on the front lip of the rear deck that come out, then pop the rear deck up and slide forward (unhook the third brake light from the trunk if applicable). The Speakers are held in with 4 8 mm bolts, be very careful on the back ones so you don't damage the rear defrost.