I'm less tolerant of kids on general principle.

But for what it's worth, you are completely within your rights, Dave. You deserve to relax, not be a volunteer babysitter. And if your friend got his panties in a knot over a polite request, IMO it's a sign of denial. If the opportunity arises, perhaps you and your brother can talk to the friend and give him a reality check on parenting before the next trip.