I understand that, but if each address holds 10$ of Bitcoin, and for each new address/input added I pay 5$ fees (because fees are proportional to the transaction weight) it is equivalent
to 5% (current fees)
There is no way you should be paying $5 for each additional input. Even assuming legacy inputs, each additional input adds around 148 vbytes of size to your transaction. Paying at 1 sat/vbyte, which is fine at current mempool levels, works out to around 7-8 cents for each additional input. If you used segwit inputs instead, this would come down to around 3-4 cents for each additional input.