No it won't slow you down, but nice smooth engagement will be compromised. If your pulley a significant amount of HP Inincrease it would help you spool to a higher rpm and engage at a much more aggressive engine speed. This will also prevent you car from rolling forward at stop lights because of higher HP. You won't need to hold the brakes down to hold back the car.

If your just running standard bolt-ons, its not going to increase performance or tranny life much at all. Your money would be better on a tranny cooler.