I've been hearing about quantum computers since the early days of the market and I'm just as skeptical and concerned as you, OP.
But you know what, after all these years, bitcoin continues to grow and become popular globally with the recognition of many organizations, companies and especially governments. Bitcoin has become a mainstream asset and could compete with assets like gold and stocks, but the threat from quantum computers remains just a rumor.
There is no real threat and it is all just rumors, so don't worry too much.