I've played both. Rockband is easier than GH3, but it has some great songs. I prefer they way GH3's fretboard looks on the screen (probably since I am more used to GH3).

If I were to pick just one, I'd pick GH3. Both are great.