[...]
I know how they work, what I don't understand is why you should send the transaction raw to an SPV server which will then send it to the Bitcoin network. Why don't you send it to the Bitcoin network in the first place?
The SPV server's job should only be to return you your balance.
They are not "servers" that run separately from the network, they are simply bitcoin full nodes that are part of the bitcoin network. When you send your transaction to a Electrum node you are sending it to the bitcoin network. The only difference is that these full nodes have an extra layer which has implemented Electrum protocol.
You can technically connect to any other regular full node and send your transaction that way but it would only complicate client implementation and your client has to perform extra steps and this whole process's contribution to your privacy is very small so it is not worth implementing.