That pretty much completely defeats the purpose of using a hardware wallet
Yep, but that fact might not be obvious to a lot of folks, and I probably would have made that mistake about a year ago before I started learning about hardware wallets
and Coinomi, which I've used in the past.
I actually like Coinomi for the Android, but I'd never keep any significant amount of crypto in it. Usually when I'm in the faucet-claiming fever (yes, I still get that even years after realizing it's a waste of time) I transfer some altcoin dust into Coinomi and that's about it. It's too bad they didn't make a completely secure wallet.
Not sure about what will transpire for OP, but hopefully he didn't lose his coins.