Difficulty will lower when miners quit.  People aren't gonna pay more just because 1000's of miners drive difficulty up.