My GingerWallet syncs to my Bitcoin Core node v28.1 and I haven't really observed any sync issues so far. Contrary to your setup I use my GingerWallet only as a hot software wallet, so no hardware wallet connected.

What is the sync counterpart in your case, an Electrum server or Bitcoin Core or Bitcoin Knots?