I'm no expert but I thought that if there are fewer mining rigs the difficulty rate automatically adjusts downwards?
Only after a period of time - quite a large period of time at that. Transaction processing is the issue here - Bitcoins can not be used if you can not process the transaction. If that is the case people will loose confidence and then it is a death spiral.
If you were trying to pay for something and ASICMiner decides they have to power down for what ever reason it almost doubles the transaction processing time. You can not have that much control in the hands of a few people - otherwise we are back to Central Banks.