Thus, one can stop all those connections via firewall rules.
Absolutely, but that's a lot more than simply connecting to your own node, which was the original statement.
Okay how do I set up my own node for Electrum desktop wallet to be used alongside my hardware Ledger Nano S?
You need to run your own Electrum server first. There are a variety to choose from - EPS, ElectrumX, electrs, Fulcrum - each with their own pros and cons. I'd agree with satscraper though, and I would also suggest Sparrow which will connect to Core directly without requiring additional server software.