You are probably doing market orders since you are being charged fees.
The thing is, in margin trading, the takers fees are calculated with your (leveraged) position value, not your margin. So your position value is way larger than your margin, and that's the basis of calculation for taker fees.