Hm... What if someone get holds of your Trezor without your knowledge, installs malicious firmware that saves your passphrase, returns it to you, then steals it again after you have used it, and downloads the pasphrase?  Or whaterver?