That's kind of my point. If we only talk about bitcoin to other people who already use bitcoin, then how is it going to grow? Long term growth of bitcoin (including sustainable price growth as opposed to speculation driven fluctuations) depends on adoption. Adoption will only increase with people who don't use bitcoin starting to use bitcoin. Merchants won't go to the effort of starting to accepting bitcoin unless they know there is a demand for it. If nobody ever talks to merchants/vendors/retailers/tradesmen/whoever about bitcoin, then they won't know there is a demand, they won't ever start accepting it, and adoption will never come.

The future of bitcoin depends on people being vocal about bitcoin and its use, particularly to people outside the "bitcoin family". I'm just not sure the best way to balance that with your own safety.