You can do that as long as you have the recovery phrase from your hardware wallet, you can use that with Electrum.
Now this means that I always have to save the copy of my private key somewhere safe at all times because the bitcoins from the hardware wallet are also not recoverable.
But this also jeopardize the safety of the hardware wallet as already there is a copy of the private key out there outside the wallet so higher chances of the funds being stolen. I guess the problem of storing the private key more securely also arises now since we know that the hardware wallet can also stop working at any time or any mishap can happen.