That's an interesting question. I operate my personal (yet non-public Fulcrum!) Electrum server for the purpose of enhanced privacy. My SPV wallets only have to talk to my own Fulcrum server, no wallet's address details are "leaked" to other Electrum servers.

E.g. I invoke my Electrum wallet with the following command line options:
The --oneserver tells my Electrum wallet to talk only to the Electrum server that's specified in the --server ... part. No other Electrum servers are discovered or contacted.

I would assume, but haven't checked myself yet, that you could specify multiple --server ... addresses to which your Electrum connects. But this needs confirmation by someone who has already tried this with success.