It's not really a stability issue..  The downtimes this morning and yesterday morning were caused by me trying to get a little extra performance out of the site by changing the frontend to use mysql_pconnect instead of mysql_connect.  Apparently these connections were not being cleaned up properly after being broken, which caused a connection leak of sorts between the web servers and the db servers.  (It wasn't an actual mysql connection leak, but more of a certain app's tcp connection limit being reached.)  I switched the webservers back to mysql_connect this morning, so that particular problem should not happen again.

The eventual fix is to move all the queries to either mysqli or PDO, as I suspect the connections were not being cleaned up properly by mysql_pconnect which, along with mysql_connect (and the entire php_mysql API) is deprecated and slated to be removed from php any minute now.

Tonight's outage was caused by something entirely different and baffling, as the web servers were all up, connected to the db, and serving pages fine, but the haproxy instance that balances the load between them was, for some reason, refusing to route the traffic.