I am not afraid of Ledger stealing the keys.
Stealing will land them in prison, so that's not my biggest worry. But leaking the keys is a real risk.
It will only land them in prison if it can be traced back to them. Otherwise it's indiscernible whether the keys leaked or an inside job happened.
But yes, andy.arden is right that in the big picture government control is probably the largest risk.