Thank you for the question - it's intended to work this way
I will quote a previous message of mine from another thread where I explained the advantages of this setup:
Well, I got your point. If I'm not mistaken, Notes are very much helpful in breaking the link, and as much time as we delay to withdraw from Notes along with the option to combine Notes and withdraw different amounts. Otherwise, using fast mode with almost instant payout, the process isn't that helpful I think since they can be identified with little effort with some explorer check, as it limits the address, and output to check.
Now, it also makes sense why having different fee would add a little bit extra privacy.