My GingerWallet syncs to my Bitcoin Core node v28.1 and I haven't really observed any sync issues so far. Contrary to your setup I use my GingerWallet only as a hot software wallet, so no hardware wallet connected.
What is the sync counterpart in your case, an Electrum server or Bitcoin Core or Bitcoin Knots?
In theory, I make sure to run bitcoin core (v28.0) to ensure that transactions sync across my node. I forgot to point this out.
I have both wallets (both created by gingerwallet and hardware wallets). However, I haven't transferred any funds to the hot wallet yet, so I'd like to test that as well.