I noticed if I pivot with my heel to push the clutch down and lift off, my shifts aren't very smooth.

I just keep my foot in the air or on the deadpedal when I don't need to shift.