The biggest worry has to be that there's a steep enough fall in the hash rate that it starts taking, say, an hour to solve a block (and potentially 4-6 weeks to adjust the difficulty).  There's not much incentive for previously sidelined hash capacity to come online because the difficulty hasn't adjusted to increase the probability of you finding a block.  Meanwhile, six hours to confirm transfers negates a major selling point and basis for demand for bitcoin.

Less extreme than this is a bit of market manipulation by the miner.  Consider a miner who assembles a rig (or collection of rigs) with, say, half the hashing power of the pre-join network (so this miner has 1/3 of the network after he joins).  He joins immediately after a difficulty adjustment and thus collects approximately 33,600 BTC in 224 hours before the difficulty adjusts.  When the difficulty adjusts, the miner cuts the power to the rigs.  With the difficulty adjustment and loss of 1/3 of the hash rate, blocks now average 15 minutes to solve, so the miner can wait 504 hours before rejoining.

Over the course of 728 hours of this strategy, had he mined continuously, he would have mined 72,800 BTC while using 728 units of electricity consumption (a ratio of 100 BTC/uec) while by stopping after the difficulty adjustment, he mines 33,600 BTC for 224 units of electricity consumption (150 BTC/uec).  Depending on the relative costs of electricity versus the cost of assembling a rig with a given hashing power, this may even be more profitable after taking depreciation into account (and remember that computing power is generally decreasing in price while energy is generally increasing in price... it may not be now that electricity is a bigger expense for a miner than depreciation but it seems a fairly safe bet that it will be at some point).