What you have in your hardwallet are not the tokens but the keys to access the tokens. Your tokens are in the contract and any change that is made in the contract will affect your tokens no matter where you store your keys. For example, if devs decide to freeze the contract for whatever reason, you will not be able to move your tokens independently of the wallet you are using.