With relatives and very, very close friends, fine. I'll be alone with my nephew. But I won't be alone with your child. I just won't.
I grew up in a place and time where homosexuality was strongly associated with sexual predators and with pedophilia, and I don't want to make you uncomfortable or suspicious or risk some sort of false accusation. I don't much care for most children anyway, so it's simply not worth it to me. It makes me uncomfortable to think that you may be uncomfortable, so I avoid the possibility entirely.
I realize that this is a more enlightened time, and that some of you, gay or straight, might find my refusal puzzling or even offensive. Don't get me wrong: I believe your child is perfectly safe with your gay friend. He would be perfectly safe with me. I'm very happy that many of you wouldn't think twice about leaving me alone with your child. But would you think once?