There is a small plastic clip that holds the rod into a hole in the handle... you have to pop the clip loose with a flathead and it turns out of the way, then the rod lifts up and out.

You do have to remove the door handle to remove the panel easily.