Someone told me not long ago: Bitcoin is a dangerous game. It is even possible for some people or groups to take over the blockchain through a majority and destroy the entire mechanism. People don't know they are playing these games.
From this sentence, you could already say why they are scared of it and its because they are treating it like a game. Bitcoin is not a game but it can be used to play games and you can win it at games but Bitcoin is a Cryptocurrency which is digital money. They should not brand something they do not know about because it could be a hindrance to other people who have an open mind into learning new things. This "Ordinary" people are scared because they have no knowledge about it and studying it or teaching them what Bitcoin and Cryptocurrencies are can be helpful to them and might even change their mind.