You can do that as long as you have the recovery phrase from your hardware wallet, you can use that with Electrum.
Now this means that I always have to save the copy of my private key somewhere safe at all times because the bitcoins from the hardware wallet are also not recoverable.
Exactly.
Actually this practice isn't just for hardware wallets but for every non-custodial wallets. Those are the wallets that give you the private keys, seed, or the recovery phrases. So you should practice that of keeping a copy of it whenever you use it to store your balance for a long time. Always create a back-up and it's safer if you won't use the cloud as a backup or any email but handwritten.
But this also jeopardize the safety of the hardware wallet as already there is a copy of the private key out there outside the wallet so higher chances of the funds being stolen. I guess the problem of storing the private key more securely also arises now since we know that the hardware wallet can also stop working at any time or any mishap can happen.
The manufacturers like Ledger don't know our private keys. Read this discussion for clarification.
https://www.reddit.com/r/ledgerwallet/comments/4xgi2t/does_ledger_know_your_private_keys/