Well if you have a Rebel of some sort now and you like it then you won't be too put off by the 350D's size. I personally have always shot with big honkin cameras. First a big honkin all-mechanical
Nikon F2, then a big honkin auto focus
Canon EOS A2, and now I'm sellin that to pick up a big honkin electro whiz bang but manual focus
Canon T90 (I want access to awesome/cheap manual focus lenses). So when I messed around with the digital Rebels in a camera store I was like, "man these Rebels are dinky." Of the two Rebels, the 300D is larger and to me, shaped better, but it is UBER plasticky. The 350D is still all plastic but it's at least a nicer plastic, but it's freakin tiny. In comparison the D70 is about the right size and although it's plastic, the plastic is nicer than either of the Rebels, and it has a proper rubber grip. It feels pretty nice.
As for the 20D, when I first looked at one I thought it would be too small but I picked it up and was floored. It's a solid hunk of metal and feels pretty damn good in my hands. The overall shape of the D70 is perhaps a bit better than the 20D, but the 20D more than makes up for its slightly too-small size in how it feels nice and substantial and of a much higher level of quality than the D70. I was pleasantly surprised. The build quality is pretty much equal to any of Canon's EOS-1 professional 35mm film or digital bodies, which is to say solid magnesium with nice thick rubber grips, as opposed to the nice-for-what-it-is plastic D70. You really feel like you're holding a substantial piece of work with the 20D, whereas the D70 feels relatively nicely weighted, but still plastic. Kind of a shame that Canon has such a huge gap in build quality from the Rebels to the 20D. They don't really have anything that's sort of on the same level of feel as the D70.