No, most people hear about silk road and wonder what it is. Then how it works. Then why the government doesn't arrest the people doing it. Then how the people involved don't get caught. Then what Bitcoin is. Then why it's worth so much. Then how they can get some. Some probably dislike and think just as low of the people buying drugs on silk road as they do of people buying drugs off silk road. I doubt many of them think that people wouldn't buy drugs online without bitcoins. People do it all the time without bitcoins already.

The real problem here is that you don't believe in Bitcoin. You think that if the Powers That Be look at the behavior of bitcoin users and don't like it, they can destroy, damage or limit it in some way.
People like to give online Poker as an example. It was booming until the Powers That Be realized money was being exchanged that wasn't under their control or regulation. So they went after it. They confiscated a lot of money and put some people in jail. And they did kill it temporarily in the U.S. It took them a few years, but it was dead. But it flourished overseas. Did so well in fact that it's back in the U.S. again.
Even if the U.S. government, or other Powers That Be, decide to make it harder for you to do business because "people like me" don't do and see things the way you think they should, bitcoin will continue. Or Litecoin or some other second or third generation altcoin will replace it. Crypto-currency is the most disruptive technology to come out since the Internet. More disruptive than smartphones, tablets, and social sites like Facebook. It's not going anywhere.
Bitcoin is still in beta. There are no guarantees that it is the "final" crypto-currency. I understand your concern over your business model, your concern that the actions of others might not help you achieve your goals for bitcoin. But your goals don't appear to match the reality of bitcoin. I believe that crypto-currency is the future. That it's already started, And nothing you believe can stop it will do so. Including "people like me." Or even "people like you. Oops, I almost forgot the picture that makes what I say meaningless.
http://www.seethatdad.com/wp-content/uploads/2013/02/temper-tantrum-girl.jpg
(grin)