What you're looking if wallet which support BIP 157[1]. AFAIK the only desktop wallet which support it is Wasabi Wallet which is controversial mainly due to surveillance on it's CoinJoin feature.
Thanks! I've read through the BIP and it seems to be indeed what I'm looking for. As Wasabi seems to be open source, maybe the relevant part can be "ported" into completely open/decentralized solutions like Electrum ...
@pooya87: Thank you; yes I've only recently looked in that file and I think I understand the way net_processing.cpp works approximately even not being fluent in C++. I still hope to not have to code that thing myself, but it seems I have to (or at least parts), so its possible I've to implement your idea

By the way, how it would work, you expect to have a wallet connected to the node which keeps only blocks which contain data about your address? How about incoming transactions, do not you want to keep blocks which confirms validity of your inputs?
Well at least Feather (afaik!) works quite simple: You set a start date for your wallet (when it was created or when the first transaction was done) and then all blocks after that one are verified (i.e. downloaded and validated) but only the important data is kept on disk. I would like to have a little bit finer-grained control, for example if I know that in some period in the middle I didn't receive nor sent transactions, I wouldn't have to download these blocks, so I don't waste bandwidth I don't need. BIP 157 seems to provide that. I think it wouldn't even be necessary to store entire blocks (those with the transactions), the relevant part of the merkle tree may be enough.
In Feather everything works smooth even downloading all blocks from the start date, something you have to wait a bit but in general it is of little issue. However, one has to take obviously into account that Bitcoin blocks are larger than Monero blocks (the difference is abysmal, XMR blocks seem to have only 15-400 kB, while BTC's are ~2-4 MB), so the verification time everytime you launch the app would be much higher.