you can't fight speculators and prevent them from coming in, in fact all the things you said (the adoption by merchants) is going to bring even more speculators in. it is because of the price and the way it moves. as it is adopted more then there will be a price rise because of the increasing demand. and as that happens we will see more newbies and profit-seekers come in and price goes even higher.
as far as we don't lose the objective altogether (which is adoption as a way of payment) we are fine, even with lots of speculators and weak hands.
People trading bitcoin for fiat is fine, but what I'm saying is that we don't need speculators and we shouldn't pretend like the people who are buying bitcoin are somehow heroes and the people who sell it villains.
If people bought gallons of milk with bitcoin and used it to just help trade gallons of milk and boxes of cookies then you'd still know the value of bitcoin without fiat being involved.
You could also make the calculation that e.g. I can get 1 gallon of milk for 0.5 mBTC... I can get 1 gallon of milk for $3.00..... Therefore 0.5 mBTC = $3.... Therefore 1 BTC = $6,000
No bitcoin exchanges or speculators necessary. All you need is a farmer who sells milk and a baker who sells cookies.
Edit: If you come by North Pinellas county, Florida I will gladly sell you 1 gallon of milk for 0.5 mBTC.
Edit2: 0.5 mBTC sound more natural as 50 millibitcents, it also sounds cooler, at least to me... You could even drop the bit... and make it 50 millicents, as no other currency offers millicents it will obviously refer to bitcoin.