Have you ever tried this? I ask because in the past I've seen many users who complained about how slow Electrum client gets when they have a much smaller number of addresses and transaction history. Like this one with 200 addresses and 1k tx.

Lookin at the code and how the wallet works, it all looks very inefficient (Electrum doesn't need to be efficient for such cases). Wallet file is a big JSON chunk that contains all the key pairs, history, etc. which I always found a bit weird as well (it seems to waste space with duplicate data), is not the best option for 10s of thousands of inputs.