I had a Ledger Nano a few years back and almost lost a lot of Satoshis because it could not handle the "dust" (small regular deposits) from my mining rigs and would never stop syncing. I was finally able to recover my sats using Electrum with my passphrase
The proper way to do this is using Electrum together with your Ledger. You use Electrum in read-only mode, make the transactions in Electrum (big plus: coin control!), and use the Ledger to securely sign each transaction.
If you want this setup, you'll need a new seed phrase in your Ledger, because the current one should be considered compromised the moment you entered it in Electrum.
The question remains can I connect Electrum to my node without using one of the Electrum servers or will it just be a standalone wallet?
You can use Electrum without running your own node. I'm not sure what exactly you're asking.