If you are dealing with the most downloaded and rated wallet apps on Google Play, you should be fine. Also, be careful because there are fake applications on Google Play.

I do not know what you mean by these wallets that are downloadable without Google. If you want to download wallet apps from outside Google, you must deal with official websites only, do not use third-party sites. I believe only a small number of official sites offer APK downloads; they rely on official stores like Google Play and the App Store.

All you have to do is adhere to the necessary security measures, install the anti-virus application, and do not download any random apps. I have been using my phone for years and I have Trust Wallet, MetaMask, and Exodus, and all the wallets are secure. If you fall victim to fraud, the blame lies with you, not with the wallets you are using in the first place.