I don't think you can do that. What you can do is transfer the token to other address.
Just transfer it to burn address like 0x000... and it will be gone from your wallet.
But why do you want to get rid of it ? You need gas fee to move the token out of your wallet.