And that is the problem - there is a reason why hardware devices are the 2nd factor - they can be lost or stolen.

Although I like the idea of a single non-password method of logging in I don't know how you are going to actually stop it from just being used by someone else without resorting back to a PIN or password again (and adding some sort of bio-scanning to the Trezor would probably not be a very financially viable option).