I am unsure whether it is due to Electrum not having UTXO's history or due to another part of the set-up requiring a direct connect.
If you don't have any unspent transactions in your offline wallet, that could be your problem right there. I've never tried to use a ledger offline, but I did experiment with Electrum and my KeepKey that way when I first got it. It worked, but I recall running into an issue that was resolved after loading the transactions into my offline wallet.