I would have to say for me the biggest concerns for bitcoin would be..
1. The Darkweb gets shutdown or takes another hit. IMO the Darkweb/illegal activity plays a huge role in the success and failure of bitcoin.
2. Banks getting involved. If Banks are losing out on money or think bitcoin is taking money out of their pocket they could create a mess for bitcoin.
3. The richest of the rich will own most bitcoins. The rich usually make money from the middle/lower class, if they own most they will be no profits for them and bitcoin become obsolete.
Just a few of my concerns and obviously just my opinion. If you have any concerns or comments I'd like to hear your input.
My biggest concerns are probably somewhat similar to yours.
I agree that bitcoin is still associated with the dark web way too much than a healthy level. People think of bitcoin and they think of the dark web immediately. This problem can't be solved by governments cracking down on dark web markets, it can only be solved by more legit people accepting bitcoin for legit things, and outweighing dark markets in volume.
Another concern would be uncertainty with the regulations surrounding bitcoin. Governments are tightening KYC controls for exchanges, which makes it a hassle for someone to buy their first bitcoins and could potentially discourage people from trying out bitcoin.
A very small concern would be the amount of scams, although you can definitely avoid 99.99% of scams if you just bothered to use your common sense.