Due to the network issues, we asked the exchanges to disable wallets. We did not see any forks being mined, but better safe than sorry.

Finally things are getting sorted. Here are the highlights:

It all started on Friday 13th...
Yes it was that type of day. I got a DM from a pool operator saying all his nodes crashed. I was thinking some sort of hardware problem, maybe the networking go all messed up. But when I looked into it a bit deeper to my surprise there was a block that was mined into the mainchain that shouldnt have been! the '833 block violated the one easy mining per notary rule, and in fact did back to back blocks. This should have been rejected, so clearly this was much more than a local hardware issue.

Since the nodes are checking each block for all the notaries, at first I could see how it even happened. So I do what I usually do when I dont know enough to fix the problem. I added printouts, lots of them to get a feel for what was going on.

Debugging is detective work. You need to remember all the key facts and come up with a theory that explained it all. One key fact was that the nodes that were crashing were usually the master branch and the nodes that kept on going without a care were from the dev (or even the bleeding edge jl777) branch. So that led me to suspect the new caching that I added to speed up the blockchain loading. Everyone is always complaining how slow it is and certainly a one hour sync is better than a 4 hour one.

Over the past months I had implemented a notaryid cache so no calculations were needed to determine which notary (if any) mined a specific block. To make things a bit more complicate, a notary can mine at ordinary diff in addition to easy diff, but the consensus rule is if a specific notary has mined a block within the last 65, he doesnt get the easy diff.

Here is what I think happened. The caching wasnt 100% reliable, so sometimes it returned slightly different values and this allowed notary 50 to mine two in a row in the non-master branches, but the master branch would reject it. I wasnt sure exactly what happened, but the proof that it happened meant that the caching needed to be fixed. I worked and worked on it, but after many hours I realized it just might not be possible to go at RAM speeds and be 100% reliable. Here is my weakness in C++ showing and I apologize. If the codebase was 100% C, I would have been able to make a 100% reliable notaryid cache.

If I cant use the caching, what to do. I didnt want to revert back to the slow master branch method... I decided to spend some time working from the bottom up as streamlined a system as possible to get the data on demand, without any caching. So now I rely on the local DB to be correct, which is a pretty good assumption as if the local DB isnt correct, you have a corrupted blockchain on your system.

The good news was this new approach was still significantly faster than the old approach. On a system without any bottlenecks (1 gbps bandwidth, 128GB RAM, 12 cores) I was getting nearly 15,000 blocks per minute in sync speed. With the old method, I would be lucky to get 3000 to 5000 blocks per minute. But the 15K was using the caching, which couldnt be used. However, I am getting double the old rate and doing a bit less than 10,000 blocks per minute, which is a full sync in 1.5 hours. Not bad!

A long Friday 13th was looking to be a good day, for a bit. The new uncached method was looking very solid and fast and I streamlined other code and since this would be a forking change it made sense to put the elected notary activation in. One update to get the new notaries active, fix bugs and make it faster.

I think I took about a 4hr nap at that point, having been up for over 30 hrs working nonstop on all the above. The unpredictable nature of the caching bug meant that anything could happen. Odds were against anything more than another fork, or crashed nodes, but those are definitely not good at all.

Good thing I woke up so quickly as there were new issues popping up. The first was a d'oh! bug where I just totally goofed up with an uninitialized variable mixing with static variables. the surprise was it was not alway messing up from that, so that needed to be fixed and pushed out. It was due to the sudden unplanned activation of notaries, originally planned for a sedate July start.

The activation of the fix passed uneventfully for 4 blocks and then a block that shouldnt be was there and a mining pool that didnt update appeared to be mining it. For a few dozen blocks it was ahead of the notarized chain, but soon the notarized chain also became the longest chain and many nodes that were on the forked chain automatically switched. As it should with bitcoin's eventual consensus.

Things were a bit of a mess as the explorers that were based on an old codebase were mostly down and the surviving .ru explorer went comatose for half hour at a time, but always came back. With vast majority of notaries on the mainchain and a (barely) functional explorer, it felt like the long Friday 13th was coming to an end.

Ha Ha. Wouldnt that have been an easy thing.

At this point I was too exhausted to do anything more than sleep for a long time, with little idea of what I would wake to. I dont remember if it was night or day.

At first it started with a small fork, just 2 blocks, not being mined. That is perfectly normal and nothing to really worry about as it usually just resolves automatically as soon as the longest chain propagates. The strange thing was that it wasnt...

Then another set of nodes gets stuck and while we could manually kick the stuck nodes, it seemed as soon as we resolve one, another node or three went off on their own. It certainly felt like this wasnt happening all by itself, the odds of such things happening are not zero, but it would be a very rare day where you get 3 small forks like that, let alone 5, which is what I think the total was of the nodes that got stuck at specific blocks.

This wasnt a money risk in and of itself as the forks were not advancing, but too much strangeness so we asked the exchanges to disable the wallets while we got to the bottom of it. Enough stress dealing with this without adding the potential monetary loss to users to the equation!

Similar to the caching issue, the stuck nodes required analyzing in detail what was going on. After a few hours of this, I had the idea to detect a stuck node and proactively make it try to get to the mainchain. this actually worked on most of the stuck nodes but it was not the right answer. More a part of the process.

I noticed that the stuck nodes would pretty quickly ban all nodes that were not on their own chain, effectively isolating themselves and without anybody mining it (diff is very high for just a few non-mining pool nodes), they get stuck and get just a block ever few hours. But the longer it goes, the longer their mini-fork gets. And why where they mutually banning each other?

I tried increasing their banscore limit, but that just flooded the nodes with massive banning errors until whatever threshold was exceeded. These nodes just did not like each other at all. After another 20+ hour shift, I finally figured out what was going on. In bitcoin blocks are rejected very early in the network message -> raw block -> validated block -> really validated block -> connected block process.

The once per 65 blocks notary rule combined with the small fork usually having a notary mined block would end up creating an illegal easy diff, which in bitcoin terms is a violation of the PoW and enough to make it want to ban a node very quickly.

Once I found the cause of the mutual banfest, it didnt take long to get a version that wasnt so hostile. I just deferred the PoW validation until the end when all the data is available to know whether it is actually valid or not. Basically it isnt possible in isolation to tell if an easy diff block is valid or not, you need all the 65 prior blocks! Such an obvious thing after you understand it.

Now I had a clean solution to the messy problem and the test version worked like a champ. All the stuck nodes I had except one, quickly caught up to the mainchain and have been locked into it ever since. The one that didnt work, well I did things to it i really shouldnt have, and when you mess with DB files in ways you arent supposed to, they break. So a full resync on that one node.

We pushed this out to the notaries and before long notarizations resumed like the usual clockwork. During the extended Friday 13th, the notaries had a hard time coming to consensus on what block they were one. This is one way you can tell at a glance if there is a possible network issue going on, just look at the recent notarization frequency.

30 hours into my "day" I had to make a set of releases. From the highly experimental jl777 branch to the conservative master branch. I got most of it right but now we have a team testing all the variations and putting it through its paces.

Oh, during this long weekend, I also merged bitcore pull request that came in, to the dev branch. Once that is fully debugged it means any komodod can directly support an insight explorer, "out of the box"

As we update the various wallets, OS, and infrastructure providers, things will get back to normal.

For the more technical, my beta branch is testing out well and it is the basis for most of the mergemaster branch which will become the new master branch after it goes through testing.

For those that need prebuilt wallets, please be patient as we get them all rereleased and if you are in doubt at all, just wait to do any transactions.

KMD is now faster, more reliable  and able to reach the eventual consensus a lot more efficiently. The only unfortunate thing is that we need to make some delays as the exchanges will take several days to all come back online