Does the difficulty determine the price of the coin? Like if less people were mining for a time would that cause the price to go up again? Or is that wishful thinking?
More the opposite. The more hardware is committed to mining, the more cost base for generation and higher price.
Amount of hardware or cost of mining has no effect on coins price. Its about how much fresh money is flowing in to buy, no matter how many are there miners.