Start using a better wallet.
I know certain wallets may automatically generate new addresses for each transaction, there are better options that provide more user control. For instance, I switched to using the Electrum wallet which lets me view all my used and unused addresses, and even select which address to send transactions from. This extra control helps me to more easily monitor all my bitcoin funds across addresses.
My 'Ledger X' has an option to use Electrum as an interface to itself. But multiplying the amount of addresses is making it messy to conclude total balance and increases amount of fee unnecessarily.
The amount of the fee does not depend on the number of addresses you use in your transaction, but on the number of UTXOs. For example, if your transaction uses 5 UTXOs from five different addresses or 5 UTXOs from a single address, the size of the transaction and thus the amount of the transaction fee will be identical.