The person with the private key is the same organizing the lottery. He could very well disappear before the extraction or simply refuse to pay the winners.
I know, I'm strictly talking cheating with the random part. In that case it would be WAY easier to just take the money than cheat with the randomness but it is possible. Perhaps someone stole the key or someone with money wanted to help his chances. I don't know. But it does have a SMALL flaw.
It is possible, but you (the lottery owner*) would need so much computing power that it would be surely more practical to simply take control of the block chain. And in that case, you'd surely wouldn't bother with the lottery.
* anybody else wouldn't be able to mount such an attack without the private key (or without breaking the asymmetric encryption scheme)