for me, it is not that bad if you are that proud to wear your Bitcoin shirt, cause in my country only a few are able to know about BTC, altcoin, trading, etc. so it really depends on the people surrounds you. Some people may not be familiar with Bitcoin, and then you may attract unwanted attention, specifically to strangers. cause they will be curious and want to discuss more about it because as I said they barely know this kind of stuff.
One thing to consider when you're wearing your bitcoin shirt is that you are potential target for theft or scams, that's because it will attract their attention and may assume that you have significant amounts of bitcoin.