If all the miners stop working, then the blockchain stop confirming Blocks, and for sure this could affect the mempool, but not sure what would happen when a miner start working after that.
That wont happen, if all current miners stop, it means that the hashrate will go down drastically that it can be mine again using GPU, to cheap GPU, and maybe to CPU again. So all those "still bitcoiners" will probably mine again bitcoin on their computers until hash rate grows again and difficulty soars and mining requires again a stronger asic to mine, it's just a cycle, it wont stop.