Is this a problem?

Not at all, if it becomes unprofitable people just turn the kit off, the difficulty gets slightly easier as less people are mining so equilibrium is restored.

Basically if everyone on earth stopped mining, then 1 person could be the only miner and it would be very easy to mine. The system works itself out naturally. There is no need to fear about miners stopping.