the reason you need to use the remove device icon is because xp is caching the drive.... you can change it so you dont have to use that icon and you can pull it out....

right click the drive.... properties... hardware... click the drive again (pic below)... properties... click the policies tab... select optomize for quick removal.