Difficulty will not go down, because network doesn't care how you mine it. Pools or solo does not matter, its number of shares per second what counts. Other point is that no one quit the pool, because it make no sence to solo mine now, unless you have couple of petahashes.