As I can reduce latency? in another Internet connection, I have 0.5s, ADSL is the problem?
On avg 10s longpool if GBT is ~3s you have fair chance to get work that is already stale :/
IMO it is your bitcoind lagging, try latest git
https://github.com/bitcoin/bitcoinCheck your bitcoind and p2pool server CPU usage, maybe it is chocking...