You're right, the high compressor efficiency would make running without an intercooler a reasonably workable proposition--but most kits out there have an intercooler and I'm of the opinion that it's still good to have one.
Here's a dyno of an LS with a Drag turbo kit that uses a T3/TO4E 54 trim.
Spool time could definitely be better, but this car is only having boost limited by the wastegate spring. The wastegate used in this kit is a Turbonetics Deltagate which can really be helped by using a boost controller to keep the boost/vacuum signal off the spring.
As for a smaller turbo, there are some kits out there that come with a T25 but they're kind of wimpy on the top end. They're potentially more responsive but they give up a lot of power potential to the T3/TO4E 54.
Ideally one would want to run a Garrett GT28RS which should reach full boost by 3000 rpm and provide similar or better peak efficiency to the T3/TO4E 54--but it costs $1200.