Yeah, the difficulty is somewhat irrelevant to determining price.
Miners will or will not mine a coin based on it's current value, or potential value (whatever they think it "could" be worth).
If it's easier for them to mine, then it gives them more incentive to mine it. Even if the value of the coin is very low.
If it's harder for them to mine, then it gives them less incentive to mine it. Even if the value of the coin is very high.
A miner can mine a coin regardless of difficulty, extremely high, or extremely low. It's a matter of what they expect to gain out of it, whether immediately, or in the future.