A quantum computer only has a quadratic advantage in cracking the hashing based wallet security.

In any case the seed phrase of your wallet should not be your main worry. A scalable quantum computer will be used to drain all utxo with known public keys, which will collapse the bitcoin price and make your wallet nearly worthless even if its specific keys are not yet cracked.