In addition to increasing the gap limit you could (1) keep track of the generated addresses in a separate database and use application logic to check for incoming / outgoing transactions or (2) send small amounts of coins just within your gap limit (e.g. if your gap limit is set to 100, you could reserve every 100th address for internal purposes and send a small token amount there to bridge the gaps).
Your wallet will need to keep track of every address starting from the first address up to the last address in your gap limit after the last address that received a transaction (that your wallet is aware of). Sending dust to every 100th address will be very expensive and will not save much in other resources because the number of fewer addresses your wallet will need to monitor will decrease only by the amount of the smaller gap limit.
Good point! The intention was less about increasing performance but more about making absolutely sure that no balances are overlooked. However you're right in that it probably makes more sense to simply increase the gap limit as necessary.
Unfortunately I couldn't find anything on how large you can sanely set a gap limit. Maybe you can set it to 100k and it works just as well.
This will be a function of how many addresses your wallet can monitor without encountering performance issues.
Are you aware of any benchmarks or do you have any practical experience regarding performance degradation caused by large gap limits? I'm genuinely curious as I'm lacking the intuition for this particular use case.
In addition to increasing the gap limit you could (1) keep track of the generated addresses in a separate database and use application logic to check for incoming / outgoing transactions
keeping a list of all generated addresses is one thing although it could be simplified to one number i.e. the index of the last address you generated. monitoring utxos is the job of the wallet.
You are of course correct about monitoring transactions being the job of the wallet, but OP was inquiring about an edge case where said monitoring may fail ie. balances in addresses beyond the gap limit.