If you do so, you trust that Google(Android) or Apple(iOS) are not evil (modifying the RNG, keylogging you...) or incompetent (allowing a rogue third-party app to do the former). If you lose your phone (or it is stolen) your money is gone. If you are scammed by the seller, it is lost.
The answer to Bitcoin's security problem is very simple: do not place all your trust in one place. A very simple example: suppose a traditional wallet on your average person's computer. One set of keys in one place. Very easy to steal. Vulnerable to drive-by exploits, trojans, social engineering attacks - etc. But we can add as many keys and factors as we need to protect a wallet. If the standard Bitcoin client included SMS security codes, it would make stealing coins extremely difficult. Now you need control over the victim's phone to move money and they can issue the final approval by inputting a code. Ultra-paranoid mode would use additional isolated devices.
Though I agree that realistically, anyone can be hacked given enough time and effort... but there does come a point where the probability vs reward isn't worth it for the attacker. Creating these conditions is trivial.
I agree, that might work. But at this point I don't see the difference with traditional metal coins and paper bills. And the risks of the previous point.
A better idea would be to use multiple redundant keys. Once again, there is no reason to put all your eggs in one basket. With the above security proposal - backup keys could be written to a USB drive or encrypted and saved to a remote server at wallet creation for use in the event that access to the phone was lost. Most of these security proposals are already in wide use.