Hardware wallets have never really been very good at dealing with large numbers of UTXOs as they pretty much all run using relatively low-end hardware with the associated memory/CPU limitations.

What might be "fine" for you average PC/desktop, can really cause issues with hardware wallets as they struggle to process/verify all the data from 10s/100s of transactions etc.