I think the reason why it slowdown even on 5 inputs may be because of the connection problem you might be using a broken USB cable
My assumption was this problem occurs while creating the transaction, but if you're right, it's while signing it. Is that correct, OP?
try to replace it with a known working one or try to clean the USB plug and terminals with a brush and thinner/alcohol.
I would not use paint thinner, that dissolves all kinds of plastic.
When I bought the Ledger back then, I thought I had made a good choice.
But now I don't have a good feeling about it anymore.
Unfortunately, what is on the ledger is not exactly small.
Still, 400 inputs isn't that much, and if that would be a problem, I would have expected much more complaints about it.
Should I rather get another hardware wallet and try to get my inventory over there?
That should work, as far as I know Trezor uses the same seed format. Whatever you do, don't enter your seed words in any website! You probably know this, but people still do it once in a while.