The reason is because moving from private key to public key uses asymmetric cryptography, while moving from public key to address uses SHA256. These differ in how much easier they would be to "break" using quantum computing.

Using Shor's algorithm, a quantum computer could reduce the number of operations required to find the private key for a specific public key by many orders of magnitude. This would allow a sufficiently powerful quantum computer to find the private key to any address which had exposed its public key, which is done whenever coins are spent from that address.

Conversely, using Grover's algorithm, the smallest number of operations needed with a quantum computer to convert a bitcoin address back to its public key is still 2¹²⁸. This number of operations is so large as to essentially be impossible.