I don't trust that myself; I'm just saying, if we assume it's actually assembled in France, and they only get bare PCBs from China, the devices can't have 'Chinese backdoors' on them. Except if they buy backdoored components of course, but as you said, everyone gets their components from China.
If we start getting paranoid about everything that can have Chinese backdoors, we shouldn't buy 90% of the things we own. That includes the monitor you will read this post on, the mouse you open the thread with, and the chair you are sitting in.
The more paranoid we are, the safer it will be for us. Relaxing and taking words on faith, we become vulnerable. Therefore, it seems to me that a certain amount of paranoia will not be superfluous. It will be better if we count and prepare for the situation (take all possible steps to minimize negative consequences) that Chinese or not-Chinese backdoors will be found in the Ledger's devices than if it turns out that we thought that everything was in order with this HW, but it contained backdoor, which caused us financial damage.
All companies lie to protect their business interests and to increase their profits. When they get caught, they will lie about why they were lying. Even when they are not lying, they are intentionally withholding the truth or parts of the truth from their users to protect themselves and their profit margins. And yes, that goes for Ledger's biggest competitor as well.
But for some reason, when device manufacturers are discussed, Ledger turns out to be the champion among them in lies. This leads to some considerations about them and about what level of paranoia the users of their products should be. Their closest competitor is Trezor, which has been around for as long as Ledger, but their lies have not been as widely publicized.