Latency actually matters ALOT. I was mining for a while on some altcoin pool. I got a whole bunch of stale's and it was because I had latency of 300ms.