As supposed your hardware usb wallet is encrypted and as such only a correct seedphrase can be use to access the address in the hardware device. If it gets stolen, destroyed or lost, as long as the seedphrase is retrievable (it was backed up) then there is no cause for alarm.
if this is the case i like it, i can store my seed words and my passphrase at separate locations
Not just any location but safe and accessible locations. Safety should be considered when choosing locatons to store your seedphrase else you might end up losing your coin due to some careless action.