Actually everyone who can listen to the traffic inbetween (including the ISP) knows that you are communicating with the server. They just don't know what is being communicated.
Then what you meant wasn't privacy, but anonymity.
If the Electrum server is actually connected to the bitcoin network (which it has to be to retrieve up-to-date information), then it is straight forward to deem that server and anyone connecting to it as a bitcoin user.
That's true if the Bitcoin node your remote pc runs isn't connected to Tor. Otherwise, they can't conclude that your remote pc runs a Bitcoin node and thus, that you're a Bitcoin user.
The main question here indeed is whether OP cares that other entities might know he has something to do with bitcoin. If the answer is yes -> Tor is required. If the answer is no -> TLS is definitely good enough.
I think we've confused that the remote PC runs both an Electrum server and a Bitcoin node. My response went to Abdussamad who I find him correct. Whether you want to remain private or not, your connection to your Electrum server doesn't matter as long as your Bitcoin node transfers information through Tor.