Does it get faster once you have less remaining funded inputs in your wallet?
I also wonder if your Ledger with the current wallet will work okay again once those inputs are consolidated, or that somehow the existence of those used addresses will still slow it down.