I had my blower motor go out on me as well. It is tough to remove like you say. I believe the key for me was to remove the two side bolts from the dash to lift it a bit and then pivot the fan motor out. These bolts are located on the side when you open the door- there are two plastic caps that you pop out and then remove the bolts underneath them. Incidentally, I have a whole blower unit out of a '90 Accord if you are interested.

Marc