One of the main advantages of hardware wallets, in addition to keeping your keys permanently offline, is that if they fall in to an attacker's hands your coins are still safe (or at least, safe for long enough for you to recover your backs ups and send them to a new wallet). Your set up misses out this important protection, unless you are also encrypting the USB drive.

In terms of the argument regarding convenience of hardware wallets versus mobile wallets, the two are not mutually exclusive. It is entirely reasonable to use both - hardware wallets for cold storage or for transporting large amounts of crypto around with you, and a mobile wallet for small amounts of crypto which you can afford to lose but it is convenient to carry around with you on a daily basis.