Only if it's REALLY cold...which is very rare. Otherwise, I can't stand wearing them to bed...or long pants (pajama bottoms, sweatpants)...or any shirt other than a plain cotton tee (or no shirt), because they all end up getting twisted around and getting tangled and stuff...pisses me off.
It has to be just boxer-briefs and a plain cotton T-shirt...nothing more, nothing less.