Always will be dangerous.
Anything can happen to the storage, the mobile phone itself, the mobile network, compromise of the private key or the numeric phrase
I don't think It's so dangerous as You're saying.I use mobile wallet for couple of year's and never faced any problems.Today's phones are More secured and Apps are much stronger on security section.So, i think using bitcoin wallet is safe.We can use coinbase wallet or blockchain and easily made transaction using our Smartphone.