We have already built quantum computers.

http://www.extremetech.com/extreme/219160-googles-quantum-computer-is-100-million-times-faster-than-a-conventional-system

Right now we can only access quantum buts in double digits.  As that technology matures, your fears may very well come true.

Personally, I feel what we need to fear is the tiny machine with just enough AI to replicate itself.  That is only a few years away.