Yea, sounds like a driver issue.

Could be another app that's trying to connect via the wireless and tanking the wireless drivers with it, such as an IM client or spyware, email, weather updates... anything like that could cause issues with drivers.