Because if you've got someone's public keys, couldn't you use those to try to derivate the master private key?
No.
Deriving child keys involves hashing the parent key with the index and then adding the public key of that hash to the parent key. The important component in this is knowing the parent key. For hardened derivation, you need to know the parent private key. For unhardned, just the public key. Either way, without knowing information about the parent key, you don't know what points or private keys were added together to form the child key. Quantum computers won't help you with that.