After watching tshark, poclbm.py output, and tailing the access log for the proxy, I have determined that a low KeepAliveTimeout setting in the Apache config was the culprit.

The KeepAliveTimeout on my server was 3 seconds, and the ask rate was 5.  This caused the miner to try to use a KeepAlive session, only to find it timed out.  Then it had to establish a new connection.  This was giving the connection issues I was seeing.

Essentially what this means is the KeepAliveTimeout setting has to be higher than the ask rate.  I would suggest adding a few seconds on top to make sure there's a little wiggle room.