well back in the days I had an 87 camry 5spd and i used to have a problem like that.
The problem was the fly wheel. They are pretty common for f'ing up.

When you start it does it make a churring/grinding noise?