I've been hearing about quantum computers since the early days of the market and I'm just as skeptical and concerned as you, OP.
But you know what, after all these years, bitcoin continues to grow and become popular globally with the recognition of many organizations, companies and especially governments. Bitcoin has become a mainstream asset and could compete with assets like gold and stocks, but the threat from quantum computers remains just a rumor.
There is no real threat and it is all just rumors, so don't worry too much.
On the one hand, it's comforting to believe that it's all rumors and nothing is going to change, but on the other hand, technology progresses extremely fast, so what might be currently true, might change in the near future. Look at the advance of AI, just within a year or two after it became mainstream, you can barely notice the difference between AI and real life in some cases.
My point is that the IT sector advances extremely quick, can we be sure that something similar won't happen with quantum computing? That's my main concern.