GPUs are of no concern. The system was ingeniously designed to adapt. What worries me is that if mining becomes unprofitable, miners will leave and the network will lose it's security. Heck, I have my biggest mining rig offline right now, and I am not the least worried.
The biggest worry has to be that there's a steep enough fall in the hash rate that it starts taking, say, an hour to solve a block (and potentially 4-6 weeks to adjust the difficulty). There's not much incentive for previously sidelined hash capacity to come online because the difficulty hasn't adjusted to increase the probability of you finding a block.
If the hashing power drops significantly enough to drag out the average block time above an hour, than not only does the odds of a particular miner catching the next block increase (even if it takes longer) but the odds of catching transactions due to backups of processing increases. So there is certainly an economic incentive for a miner on the sidelines to jump in if some major miners were to suddenly drop out, even before the difficulty is adjusted.