(latest git with minrelaytxfee set to 0.005)
I might have been mistaken: the git HEAD version uses satoshis for this parameter (which means it probably is faster than 0.8.1 even with defaults parameters). I converted it to satoshis and my latency is back to <0.10s.
Scratch that, Prattler analysed the bitcoin sources and I was right initially (which means my very good latencies were because my bitcoind didn't include any transactions in its memory pool...).