If 50% of miners quit because of difficulty, supply will go down 50% and difficulty will go down LATER to make supply catch up.

Just look at the graphs right now - as soon as there is less hash rate than what the difficulty suggests, you'll see a drop in supply too. Also since the network grew quit a lot so far, even a "correct" block rate of 6 blocks/hour would be quite a cut in supply!

Yes, it can be argued, that if 1 Bitcoin is worth 1 Million USD, it will still be worth to mine with very high difficulties and just get a few Microcoins.
You also need to factor in the psychological aspects of mining - it is much nicer to get 10 BTC worth 10 USD each than one single Millibitcent (0.001 BTC)  when 1 BTC is worth 100 000 USD. People are used to 2 digits max. after the decimal seperator when it comes to currencies. Yes, we could then just switch to trading in Millibitcents or so - BUT this is currently not happening, so people are more reluctant to spend 50 USD for 1 BTC than 50 USD for 1000 Millibitcents!

Currently for an average miner, you only get "a few cents" per day. Even if these would be worth more than a dozen coins some time ago, it sounds less, so they are being valued less. Even right now with >15 USD/coin I rarely see "uneven" amounts of Bitcoins asked/offered in the marketplace for example!