If you loose your cash wallet what happens? Finder of it steals your money.
If you loose / get stolen your hardware wallet; "some people may steal your money" (with 0.000001% chance).
Till the thief extract your priv keys; you can transfer them somewhere safe.

So; it's still safer than fiat.