Not sure what went down, something crashed bitcoind, which is unusual on my server.
The congested mempool that we've had recently from the 14kB blocks is causing excessive memory consumption. From my investigation so far, it looks like there's a memory leak in getblocktemplate/CreateNewBlock. This makes bitcoind's memory consumption increase by as much as 1 GB per day. I've been restarting my bitcoind processes every 2 days or so.
We keep relatively strict limits for bitcoind's mempool on the node, granted below is only since this mornings restart, I'll keep an eye on it.
bitcoin-cli getmempoolinfo
{
"size" : 1967,
"bytes" : 1745856
}
I'm inclined to believe the memory leak... although I've only seen it on my Linux (Ubuntu 14.04) node. I watched bitcoind spool up to consume 8GB of RAM before it was shut down. On my Mac, using the same codebase, I regularly only see about 400MB of RAM being consumed by the bitcoind. Both Linux and Mac are built from source.
Anyway, back on topic... I wonder why my reject rate is always so high mining on your node windpath. My ping time to your server is about 17ms, yet I'm constantly seeing 10% or higher DOA. When I was running my own VPS node (with about the same ping time), I would typically see about 3-4%. Yes, running my own local node when I'm at home is considerably lower (usually about 1% or less), and that's to be expected since my miners and my node are all hardwired into the same internal gigabit network.
I'm wondering if it is caused by the traffic on the node, and p2pool's inherent single-threaded execution. Anyone have thoughts on it?