I had a very similar problem in my old Volvo 240 - the soldering inside the signal relay was cracked and old, and when the weather changed, it would warp the shape of the solder, forcing the cracks even wider, and breaking the electrical connection. Once things warm up, the gaps get smaller as the metal expands, and it works again. I guess one of your connections is worse than the other, since it takes longer for one to come on than the other. Volvos are infamous for their terrible, terrible, TERRIBLE electrical systems (need I say "poorly insulated, spontaneously melting wiring harnesses"?), but I've never heard of Honda having this problem.

There are two solutions, one cheap and one REAL cheap if this is, in fact, your problem. Solution A is to just buy another flasher relay - they are dirt cheap (less than $5 at Pep Boys), and easy as pie to install. Most aftermarket ones will last the remainder of the life of the car. Solution B is to remove the existing relay, open it (easy to do), and resolder the connections. This is what I did on the Volvo - it's free, easy, and works every time.

Hope this helps! Good luck