Unfortunately, you are right.
but, if you only exchange bitcoin for your private purpose, without gambling or illegal trades involved, i don't see problem here.
I mean, you should have right and freedom to spend your own money in the way you want, if you don't break the law.
Gambling and illegal trades can be illegal but bitcoin shouldn't.
But if PayPal or any other recognized payment processor adopts bitcoin as their virtual currency, it would be safer for one to transact in bitcoins as one would need to verify their account and then use bitcoins. They would have to make and receive payments only via these recognized payment processors and thus, bitcoins will not be used for illegal transactions.