Well heres the story

I went over to a friends house (car was just fine, no battery light or anything) and came out about 4 hours later (like 30-40 degrees outside) tried to start my car, turned the key and all the lights came on just fine and went to start it and all I got was this clicking sound. I tried it a couple more times and that was still all I got. I then looked under the hood, cleaned off the positive battery terminal (had alot of blue junk on it) tried it again and still just the clicking, now i'm really frustrated, went back and cleaned the terminal some more and jiggled the positive cable a little, went back to my car, and turned the key, after like 2 seconds of clicking the starter kicked in and the car started, not wanting to tempt fate I drove home and when I got there I tried starting it a few more times and it worked just fine. the strange thing was the clock in the dash was reading one o'clock and I usually keep this thing set perfectly and it was around 11:30. Any ideas on what this is and if I should be worried?

THX!