Be careful with sweeping without storing the private key: you might have to prove ownership of the private keys by signing a message.
Example for claiming refunds or requesting a transfer of ownership to another address.