yes you are right, if the hardware wallet broken, you will lose all of your bitcoin, so there are many people that use hardware wallet for investing in a very very long time, so they dont use it everyday, and every time,
if you want to use it everytime i think the software wallet is a better choice than hardware wallet