One obviously should encrypt the wallet file if it stores sensitive information (i.e. private keys / seed), but without sensitive information being stored (which is the case when using electrum with a hardware wallet) that's definitely not necessary.
Worth just pointing out - if you don't encrypt your wallet file, it can be opened in watch only mode without your hardware wallet. So if other people have access to your device, or if they somehow gained access to your wallet file, they would be able to see all your addresses, extract your master public key, and see your balances. Not encrypting it isn't a
security risk (they can't spend or transact any of your coins), but it is a
privacy risk.
The downside to encrypting it is of course that you can't open it as a watch only wallet.
OP has his wallet protected with a password.
Therefore without knowing the password, other people can't simply open the wallet and spy on his transactions.