My rubber part was really stuck too. I just pulled at it very hard, and twisted a bit. Underneath it, the metal brake handle is pieces of metal put together, which seemed to be glued on too. A littler persistance paid off, and after 5 minutes of pulling and twisting I got it off. If you are really brave, you might even want to cut all the way up the rubber you are removing, and just peel it off.