If you have those types of skills I assume you're skilled enough to set up a Linux machine with a full core node and your own SPV server. (...)
Exactly. And having your own Electrum server you may use it not only with electrum, but also with Blue Wallet on your mobile, Trezor Suite etc.
By the way, how it would work, you expect to have a wallet connected to the node which keeps only blocks which contain data about your address? How about incoming transactions, do not you want to keep blocks which confirms validity of your inputs? How about the current state, if you want to create outgoing transaction you plan to keep the block and then download/ignore blocks which does not have information important for you?