Which link?
Miners to p2pool
bitcoind to world
I have run the miners locally on the Mac Mini and both bitcoind and p2pool, that did seem to make a difference on the timeouts.
From forrestv response, the ReplyMatcher timeouts are between bitcoind and p2pool. So WiFi should not effect, I guess.
My latency and efficiency seem within norms so may be the ReplyMatcher timeouts are just noise.
I would get rid of all of it & go wired. Mining over wifi is notorious for causing errors/problems, it's the weakest link & by removing it you are eliminating it as the cause of your problem.
Though if you have to choose one, I would definitely get the p2pool host on ethernet. Both would be best, but it's slightly less important for the miners.