(The reason I use a deck of cards in addition to a die is to guard against potential bias in the die.)
Just roll the dice a few more times :-)
An honest die provides log2(6) bits of entropy, that is 2.58 bits.
A pretty biased one, where the odds of getting 6 is twice that of any other number (2/7 versus 1/7 for the five other possibilities) gives you
S = - SUM_i p_i log2 p_i = - ( (2/7) log2(2/7) + (5/7) log2(1/7) = 2.26 bits
Not much less, and that is a pretty biased die. If you roll your die 100 times, you get 226 bits of entropy; more than enough since the work of breaking the discrete logarithm problem with a 256 bit key is only 128 bits. But you could just roll it 113 times instead.
TL;DR: Even a biased die is an excellent source of randomness.