But trezor claim that the labeling file is encrypted with a private key derived from the trezor device is this still safe
I highly doubt your Bitcoin will be lost if you lose this file. Even after reading the link provided by Husna QA from Trezor directly, there are no notes or warnings about the risks of losing this file.
So as I previously said, the only purpose of this file is to simply organize your Bitcoin in a better way by labeling Addresses, Transactions et cetera.
If you want to be 100 percent sure and have no more fear or anxiety about it, you can try to create a new Wallet, enable Labeling on it, move a few Dollars worth of Bitcoin to it, disable Labeling and try to spend it back.