But then why does it need to redownload everything to rescan an address ?
Full nodes that need to rescan the entire chain to check the balance of a new address don't keep the entire UTXO set. Remember that you should keep only your own UTXOs if you're using Bitcoin for personal use. You only need the UTXOs when you want to analyze the chain or use commands such as
getrawtransaction.
I suspect that if you set
txindex=1, you won't have to rescan and wait for so long, whether you're pruning or not.