What version of Electrum are you using? I've heard that some versions are quite buggy with the Trezor so if you're running an older version I'd suggest updating to 2.3.2 and seeing if that helps at all.