I would suggest rather having more backup but this could be a problem for many as storing them on various places is hard to remember and what if someone got access to it by mistake
If you can't even remember visiting a single location which isn't your house and physically hiding a copy of your seed phrase there, then you've probably got bigger problems to worry about.
As I said above, if you are worried about someone gaining access to it then you need to first look for a better location, but you can also consider using an additional passphrase or a multi-sig set up so even if one of your back ups is compromised your coins cannot be accessed.
Can you tell me which HW wallet allows that?
I know that ColdCard and Passport both give the option to display the seed phrase once you've unlocked the wallet.