We are well aware that polls show that voters would trust gays or Muslims over atheists. I've been in conversations with other parents where it was stated that they wouldn't let their child play at so-and-so's house because the parents were atheists (not realizing I too am an atheist).
People aren't worried about proselytizing, they're worried about amorality. Those without faith are assumed to be amoral. Coming out as an atheist is a bit like revealing to many communities that you're a sociopath - it's done with great care if you don't wish to be ostracized.
This is also my experience: religious believers are uncomfortable with unbelief because they think that unbelief equals immorality. Many believers seem to think that without God there is no reason to be good, so they assume that unbelievers will of course do every sort of evil thing. This is, I think, the root of the sentiment one commonly hears in America, that all religions are essentially the same, or equally good -- as long as you acknowledges that good and evil are part of the fabric of the cosmos, and that it is our human study to side with good, you are welcome in society.
I find this quite strange. My reading of history suggests that sincere believers are capable of every sort of evil, large and small, and I have personally found lots of non-godly reasons to be good. In fact I am not tempted by most forms of evil: money, power, liquor, drugs, and casual sex just don't interest me very much. I cannot see why anyone needs god to tell him not to be a drug addict, a drunk, a gangster, or a bully -- those things are bad in themselves and they make people miserable.
Following the Buddha, I find that compassion is the key to morality. The Golden Rule is a valuable teaching whether or not it was taught by prophets. I find that the more positive energy I invest in relationships, and the more I control negative emotions, the better I get along with everyone and the happier I am. Why that has to involve god, I have no idea.