The reason is because moving from private key to public key uses asymmetric cryptography, while moving from public key to address uses SHA256. These differ in how much easier they would be to "break" using quantum computing.
Using Shor's algorithm, a quantum computer could reduce the number of operations required to find the private key for a specific public key by many orders of magnitude. This would allow a sufficiently powerful quantum computer to find the private key to any address which had exposed its public key, which is done whenever coins are spent from that address.
Conversely, using Grover's algorithm, the smallest number of operations needed with a quantum computer to convert a bitcoin address back to its public key is still 2128. This number of operations is so large as to essentially be impossible.
And add to that, that quantum computers would be much slower in going through 2
128 operations than a "traditional" computer.
Quantum computer's magic lies elsewhere. With Shor's algorithm QC can "see" the right private key from public key without going through all the possibilities. But with SHA256 it can't do the same.