Your set up misses out this important protection, unless you are also encrypting the USB drive.
If the USB drive is a live OS and the private keys or seed are not saved in persistent storage, the solution can work.
If the user also makes sure this setup stays always offline, it can even work as cold storage.