Difficulty will only drop if a lot of farms turn off their gear. It's likely that most folks will be mining at a loss for the first period after the halving. What sucks is, since mining only alters the total supply by a fraction at a time, the supply/demand curve won't be moved very much at all by the halving so it won't have much effect on the exchange rate.