If the privacy is your major concern then you have an option to run an Electrum personal server and benefit from its use.
Everyone should be concerned with privacy when it comes to cryptocurrency, but running a personal server is not something that anyone can do. My questions is just about possibility that any owner of Electrum server can see our IP address and our coins address/es. In today's world where hackers are becoming more and more imaginative, the use of data from such servers can serve them for target selection.
It would be ideal if everyone could have a personal server, or if everyone could use Tor, but most still do not hide their true IP address.