This simple scenario presented by the OP shows us how little many people know about probabilities, and there are studies which corroborate this, after all even if the odds of the dice landing on the number in which we placed our bet are still the same, once the payout doubles it is worth to always take the bet as you are now the one playing with an advantage, and yet there are many people that would not take that bet as anything below a 100% win rate seems to be too risky for them.
Well, as per the scenario laid out in the OP, you can only roll the dye once. The probability of winning is therefore only 17% and of losing 83%. So there's also a logic behind not taking such bet.