How do you set this $100 limit? Do you understand that what cost today $100, tomorrow might cost $1 or $10,000? As long as user is ready to pay the fee, he might trade (in our case transfer) tokens that even worth 0.

Everyone that is interested in trading must ethereum gas tracker https://etherscan.io/gastracker and catch low gwei. Usually they are low at night and in the morning till 10:00. Current transaction fee of ~60 gwei or $1.80-1.90 seems low.

And dont confuse where you send and which wallet you use. If you try to send to uniswap, get ready to pay $10 fee, while when transaction I've send from myetherwallet to binance today cost me around $2,10 only.