Quote from: Biodom on December 25, 2015, 06:35:09 PM

Quote from: wizkid057 on December 24, 2015, 07:33:15 PM

Quote from: Biodom on December 24, 2015, 06:35:43 PM

Quote from: wizkid057 on December 09, 2015, 05:38:40 PM

Greetings,
TLDR: There was a DDoS along with some data center network issues last night. All is well now. No Eligius servers were compromised.

well, that's not all, right?
we lost a block due to...things, which may or may not be directly related to ddos.
did not exactly get the point why the block was not submitted on a timely basis ("because of extreme latency at the datacenter").
i don't think that the datacenter was ddosed, right? maybe you need to be at a better datacenter?
My personal loss ~$25
oh, well

So, I publicly post a detailed explanation of exactly what transpired that day (over two weeks ago now) with to-the-minute timestamps of everything that happened... and that's not good enough for you?

Exactly what I posted is exactly what happened. The loss of primary connectivity at the datacenter caused what would have normally been an easily filtered DDoS attack to severely saturate secondary connectivity for the entire datacenter with ~~hundreds~~ thousands of machines competing for tiny slices of remaining bandwidth with latency as high as two minutes for a short period. Nothing else that could have been done to prevent this. Aside from putting in my own 4G LTE backup connection or having some alternate connectivity for the block submission paths in the future there would be nothing to be done to fix against this.

I was not trying to scald you, just pointing out that TLDR portion was incomplete without stating a fact that one block was lost.
Yes, it was stated in the long version as it being "stale". I did not see a statement like this: TLDR: was ddosed, lost one block, instead it was: pool/datacenter was ddosed, all is good now, nothing was compromised...hence my original post.

Essentially, apart from commenting I have no other issues. I don't come to this thread often, so again, apologies for bringing something of two weeks ago-this shows simply when i last looked at the thread than anything else. I am sorry that this caused some distress, which was NOT intended.

It was a TLDR version for a reason. I just won't post a summary anymore and there won't be any problems.

I recall the question posed as to why I was moving things to a new data center, but can't seem to find it to quote. The new data center is run by the same company as the old one, basically... but is physically closer to my location so it's easier for me to work on things when I need physical access to the hardware. Plus the new data center has much more capacity than the current one, which has pretty much reached the limits of physical space for hardware. That said, physical proximity came in handy last night.

As for news, at about 3:45AM my time this morning I was alerted to a connectivity loss of one of the new Eligius servers that was handling some of the pool's stratum traffic. (There are multiple, and the others were picking up the slack for the most part). It wouldn't respond at all via the network or KVM, and remote power cycling wasn't bringing it back online. I headed over to the data center and got there at about 4:15 AM (it's only about 15-20 minutes from me... closer to 15 at 4AM on a Sunday morning) and investigated. Power supply in the machine wasn't working, so swapped it out with a spare from the data center's stock, and all was well. At about 5:30AM I was back in bed.

Miners were mostly unaffected aside from a connection drop and reconnect around the time the one server went offline.

I much prefer my own hands to some NOC tech's grubby remote hands, not that they've ever been a problem. Wink

The staff at the new data center doesn't even have a key to the rack that has Eligius's servers while I'm in town (which is most of the time).

Anyway, all servers are online and functioning normally and I'm slowly continuing the longest migration in history to the new data center. I'm going to keep a couple of servers online at the original data center for backup purposes for as long as they allow me to do so. Mainly I've been working to make sure mining is mostly unaffected by the whole migration process.

Happy Holidays!