I think one of the biggest problems is security. There is a lot of comfort on the market. An arrangement related to this can be carried forward but without restriction, it can move forward this sector.
Security at what source? Your actual coins or third parties like exchanges and payment processors? People are responsible for their own money with bitcoin, but it's not like fiat and it's banking systems don't have issues with security. Banks and personal accounts are always being hacked and leaking personal data.
The first biggest problem is the scammers. Which in every way will break into the system. The second main problem in my opinion. These are governments and their bans.
Most governments won't ban it. It would be counter-productive for them because they'll be missing out in millions from lost revenue taxes. They'll regulate it though, but that's to be expected especially when they want their tax money.