For me it actually seemed like it was easier to lose weight then gain. Just working out for 3 years (plus prolly induced some growth spurts) I went from like 160 - 220... plus prolly like 5" - 6" inches. It wasnt all muscle though, I was kinda chubby.
But I went down to like 180 over the summer... and the muscle is still there.