Yeah I think personal greed takes over. If everyone agrees and quits mining, and one person continues, he is one lucky bastard the next day because he just got a lot of bitcoins in one day, lol.
Actually, if all the miners quit mining, but one person continues, they would get approximately the same amount of bitcoins as they normally would have gotten. The difficulty doesn't change, so that miner would still have just as hard of a time solving a block as ever. If he were part of a pool, he likely wouldn't get any coins (but he would get shares).
If everyone quits just before the difficulty change he would get bitcoins quicker. I think difficulty reductions are limited to 25% or something.