Quote from: Kazuldur on July 27, 2016, 04:27:09 PM

We'll prepare a more official post-mortem post in a few days. It'll sum up what exactly went wrong both before and during the incident and what we're planning to do to prevent it in the future.

However here's a quick summary of what happened today:

1. the initial backup restoration finished at about 9:00 UTC
2. upon inspection we discovered the backup was broken. One of tables crashed due to running out of disk space and our backup tool just silently skipped this table and all tables after that. It included: multiple not-important cache/log tables, configuration of Safety Limits, our outgoing transactions, users (emails, passwords etc), audit of login attempts, configuration of ACLs.
3. to restore missing tables, we recovered an earlier backup. That means that changes made to these tables in up to 2 hours before our yesterday's crash could be lost. If you did any changes, please verify that they're still present (especially Safety Limits, changed passwords or ACLs).
4. after restoring missing tables from older backup, we need to rebuilt all caches. That took about an hour.
5. we made sure that everything looks correct and we enabled access to FaucetBOX.com website. In case we missed any inconsistencies, we waited before enabling sends from faucets.
6. we reenabled sends from faucets.
7. we'll skip processing payouts today, just in case there's still something we've missed. If we don't find any errors, payouts should be processed tomorrow evening. If you see any inconsistencies (missing deposits, too much / too low balance), please let me know through PM here.

Weird trivia: currently we're processing 10-20% more sends per second than usual. Looks like users didn't run away yet

THANKS!!!! WORKING AGAIN!!!! Grin