Sounds like he's trying to take advantage of your situation.

Noone drives my car except family members.