If you are setting your gas higher than suggested that is if you are trying to do a normal transaction which uses gas of 21000 with placing a gas of 100000, then you need to pay a higher transaction fee than expected. Higher volume of gas are used only when you try to transfer your tokens, otherwise you could use a standard gas limit of 21000. Gas price/ gas matters to make a faster transaction, otherwise specifying higher gas just costs you a lot.

Never use lower gas like around 21000 when you are trying to transfer your tokens. This will make gas to be burned and your transaction will not be confirmed and you will end up with a loss in transaction fee.