What happens if the N PoS "miners" aren't online? That seems quite likely very often.
Let's say N=3 and only 10% of all stake is online. Suppose our target is to generate 1 block in 10 minutes on average.
Then PoW miners need to generate 1000 valid block headers in 10 minutes. Then, statistically, one of block headers will have all 3 stakeholders online.
So, basically, N=3 and 10% of active stake corresponds to 1000 times lower PoW difficulty.