I think part of the problem is Pontiac's brand image. The images of Toyota conjour up high resale value, reliability and quality. Pontiac, just like the rest of GM, has suffered in all three of those areas for nearly thirty years.
It's Pontiac's own fault, if you ask me. Personally, I think the Vibe looks better, but neither of them look particularly good. Their engine choices aren't that great either, even though they may look good on paper.
I have seen many Matrixes' (Matrixi?), but have yet to spot a Vibe on the street.