Any wallet needs some information about the current state of the chain.
Bitcoin Core builds that information from the blockchain. From all of it.
In prune mode it still downloads it all to create the information it needs.

If this is a problem, you better use a simple wallet like Electrum, which (from what I know) relies on 3rd party web sites to get the information it needs.