We do need to remember that processing power of quantum computers can increase much more rapidly than we are accustomed to with classical computers.
With each extra bit, a classical computer has more possibilities, but can still only process one at a time. The difference with a quantum computer is the quantum superposition of states - all possible options can be tried simultaneously - so each time you add an extra qubit, the processing power doubles. 1 qubit - 2 states, 2 qubits 4 states, 3qubit 8 states and so on, effectively 2^n classical processors running in parallel.
But having said this I do agree that a QC that is a threat to bitcoin is likely some time away - it's just that the problems are more to do with phenomena such as decoherence, error rates and a near-absolute-zero temperature constraint, rather than, as we might think, the number of qubits. It's more single problems to be overcome than it is scaling up the processing power. Tremendously challenging problems, yes, but a different sort of problem to what we're used to thinking of in terms of computer advancement.

I don't think this is an issue. The algorithm already exists. Whilst a classical computer would take an unimaginably huge 2^128 operations to derive a bitcoin private key, with a QC running Shor this becomes a much more manageable 128^3.