It's the same when using a passphrase. When using a passphrase of sufficient length and random characters (which everyone should be doing) it becomes quite time consuming and a real pain to enter it when needed. There are only two other options though: Enter it on your computer like you do with Trezor, which can expose you to key logging malware, or create a hardware wallet with a full keyboard, which would be both bulky and more expensive.

It's not great in terms of usability, but at least it's secure.