Quote from: butjust41day on March 13, 2013, 02:09:32 PM

Quote from: burnside on March 12, 2013, 08:36:48 PM

You're definitely right, there is another ongoing issue. That is that pushpool gets overloaded periodically. (a few seconds every few minutes.) I've expanded file descriptors and every single restriction I can think of to work around this, but have been unsuccessful.

So, unable to expand pushpool I ended up spinning up two pushpools and load balance them using nginx. I made that change ~6 months ago. Now we're back to one pushpool or the other getting overloaded periodically, and with the load balancing what happens is the overloaded pushpool gets removed from the balancing and you get sent to the pushpool that is still answering. If you're making a request to submit work, and that work came from the other pushpool, then the current pushpool doesn't recognize it and it gets flagged as invalid.

Oddly enough, pushpool has memcache functionality and both pushpools are pointed at the same memcache. I thought initially this was so that you could run a bunch of them and have them share the work between them, but clearly that is not the case. I'm not really sure what pushpool is using memcache for.

As you digest all this you're probably wondering why then can't I just add a third pushpool to the balancing. The problem is that in order to make sure that you always get sent to the same backend pushpool (because of the issue where your work is invalid if you don't) I had to configure the balancing to be by IP address. And, naturally, since we're behind a DDoS service, 80% of our traffic comes from... the same IP address. Ugh. So even with the two pushpools, one takes like 80% of the traffic and I have no way to split it out beyond that. I need like 3 DDoS services with each one running a pushpool behind 'em. Wink

Firstly things do seem to be running much smoother this morning.

Thank you for your explanation above. I have to admit I had to read it twice; but, your explanation is very clear. If pushpool keeps a running log of what work it has issued and that log is not being shared in the memcache where is it? Do both running pushpools share a common database where information about each workers contributing shares ect. are maintained? And is the log of work issued also apart of that database? If it was might this be a database issue?

One other thing if the memcache is not for sharing of information of what work has been issued should both pushpools be pointing at the same cache or should they have seperate memory blocks?

Forgive my ignorance here because I really don't know. I've just found in my line of work it's good to bounce ideas off each other and sometimes even a wrong idea triggers a right solution.

Edit: I don't know if this really means much; but, I notice something today when I was monitoring the situation. When I went to the My Stats tab it seemed to hang up. While it was hanging I checked one of my rigs and the unknown work rejects we're happening. I had been running at below 0.5 percent stales up to that point (suddenly I was at 20 percent; although my miner calls them unknown work). The My Stats tab has to access the database to display. Interesting coincidence?

The work is stored in pushpool's memory I'm pretty sure. It's all internal until a share is submitted, then the result of that share (stale or not) plus the username submitting the share is submitted to the db as an insert. I use insert delayed even, so the db would have to be completely down for it to impact work submissions. Edit... probably worth noting here too that the db server is a different box. We have three servers right now, the webserver, db server, and pushpool/litecoind server.

The other possibility though is that there are network issues or rate limiting going on at the DDoS provider level. Eg, if you can't load the site and the shares delay on submission, then your requests might not even be getting to us.

I'm tempted to just turn off the DDoS protection, but I know as soon as I do we're gonna get caught with our pants down again. Wink

(for those of you new to LTC, we have been one of the few pools that have been up through most of the DDoS attacks, though it does impact us too.)