Arguably, the difficulty is being driven unnecessarily fast because of miner speculation on the existing subsidy, and some eventual decline in the hash rate would be fine and expected. Whether it's an orderly decline -- as opposed to a downward spiraling crash -- is what matters.
The problem is, we have no idea how much fee pressure is enough to strike a balance between sustainable mining rewards and the diminishing returns from mining speculation. Segwit's 4MB limit may already be pushing the upper bounds since we can't just rely on mass adoption. We need to make the system sustainable with or without an exponential increase in user base.
Early miners wouldn't have been able to conceive of the figures flying around. I do wonder how it's going to balance itself out. I presume mining machines will get more powerful and efficient and miners will seek out cleaner and cheaper power. There'll always be money to be made and I can't ever see deep pockets vacating in favour of the little guy.
It's the biggest unknown looming but I guess it always has been and every doomy scenario so far has been shat upon in an emphatic fashion.