Assuming miners are rational (which they're not), then there will always be "just enough" miners to keep the difficulty at a profitable level - even when there is no block reward.