I usually only take them off if my feet get particularly swampy.
I stopped into the office last saturday coming straight in from a camping trip. I was covered in dirt and reeked of camp fire, and my socks were trashed so I walked around barefoot. Luckily hardly anyone was in the office, but my new receptionist did look at me kinda funny.