i make it clear, that downpayments are non refundable.

if you said it's non refundable, then you owe him nothing.

but if you didn't make it clear, then it's half your fault and half his, so i'd give back half of it.