As long as I keep the device safe, my identity is safe.
And that is the problem - there is a reason why hardware devices are the 2nd factor - they can be lost or stolen.
Although I like the idea of a single non-password method of logging in I don't know how you are going to actually stop it from just being used by someone else without resorting back to a PIN or password again (and adding some sort of bio-scanning to the Trezor would probably not be a very financially viable option).
From the FAQ:
If somebody steals my Trezor, they'll just empty out my wallet before I have the chance to restore anyway. Right?
Not at all. All operations on TREZOR require the user to enter a PIN and a one-time password (OTP) . The attacker would have to guess your PIN which is very difficult because the one-time password makes brute-force attack almost impossible.
So it appears secure even if I lose it! I just have to get another Trezor, restore the seed I wrote down to the new Trezor, empty it, and then reset the Trezor with a new seed and fund the Trezor.