It's the same problem with every other currency - USD, GBP or whatnot. There will always be speculators. There will always be traders. There will always be people who actually use bitcoin as a currency.
Just let it be...
Speculations affects the price of bitcoin too so with supply and demand. If all are holding,there is a possibility of bitcoin to rise because the demand is high or there is a possiblity of another coin to rise because the bitcoin users will have to used the alternative coin. That is only a possible scenario.