OK, eventually I ended up removing the blocks and chainstate directories, and recovering from a backup I made in April. Re-syncing from there took a while, but it worked. Apparently, the issue was a rare kind of glitch on my system; I'll be making more frequent block chain back-ups in the future.

OK, not much help from the core dump here. Apparently, the failure happens in a background thread that originates from the levelDB code, not from the Bitcoin core code. So it could be related to any already-loaded LevelDB database.

Now, while I'll do some further debugging, does someone know where to find the database lay-out used in Bitcoin's LevelDB databases? I'd like to know how I can recognize inconsistent states, and recover Bitcoin to a consistent state.


EDIT:
Apparently, .bitcoin/blocks/index/LOG gets rewritten after each attempt to start bitcoin-qt. The most recent contents right now is:
Interestingly, there is a 053994.ldb in the directory. So, my current guess is the error is in that file.

EDIT 2: apparently, every time I re-start bitcoin-qt, it creates a couple of new .ldb files in that directory. I don't know yet whether that's a good or a bad thing though.

I'm having the same error right now, with Bitcoin-qt 0.9.3. After downloading and processing several weeks of new blocks, it crashed, and now it doesn't start anymore:


debug.log:

I don't think I have bad RAM, but  guess it doesn't hurt to do a RAM test. Anyway, the error now happens consistently a few seconds after start-up of bitcoin-qt, and I think it's unlikely it would hit the same RAM error each time in the same way. So, even if it was initially caused by a RAM error, right now it seems something is wrong with the state of the (block chain?) data on my disk, which is blocking me from re-starting bitcoin-qt.

Can I draw the conclusion, based on the log lines "Opened LevelDB successfully", that the error is NOT in .bitcoin/blocks/index and NOT in .bitcoin/chainstate? Based on the source file that reported the error, I'd say it has to be in something LevelDB-related. What does Bitcoin do AFTER successfully opening .bitcoin/chainstate?

I'll see if I can make a core dump and examine the function call stack. In the mean time, please reply if you know something that could help. My objective right now is to try to repair the files on my disk (e.g. removing the last ones that might be corrupt), in the hope that I can recover from this situation without re-processing the entire block chain again.

Or you can check it the stupid way: make a lock time which is in the past when interpreted as little endian, but in the future when interpreted as big endian. Then make two transactions, which spend the same output, but with opposite endianness of the lock time. Broadcast both, and check which one ends up in the block chain.

Beware that if you make any mistakes, you might draw the wrong conclusion from such a test. For instance, don't forget to make one of the input sequence numbers non-equal to UINT_MAX, and don't forget, when choosing your lock time values, that there are two ranges of lock time values: one part is block height, and the other part is timestamp.

What you can also do is search the block chain for existing transactions that use the lock time, and analyze these.

Since you mentioned complexity and Einstein, the following Einstein quote is mandatory:
Everything should be made as simple as possible, but not simpler

In hindsight, some things in Bitcoin could have been made more simple; for instance, the way how transaction signing is performed (which, BTW, leads to the current transaction malleability problem). This shows that, even though Satoshi can be called a genius, not even he is perfect.

If you want to become an expert, it doesn't hurt to be a programmer. If you have no programming experience, I think Python is a nice language to start with: it's relatively friendly to beginners, and it has everything experts want (except, arguably, speed, but often that's not critical). Then, you might want to check out my implementation of Bitcoin transaction algorithms in Python. Calculation of the transaction hash ( = transaction ID) is a single line of code in the getTransactionID method.

Not that I am aware of.

E (the escrow service) is not really a hub: as long as parties can settle their conflicts without involvement of E, E is not involved in any of the transactions (this is a major advantage in terms of privacy and liberty). Also, there is no inherent, architectural reason why escrow servicing should be centralized to a single entity: in theory, every channel in the network can use a different escrow service. However, in practice, due to the difficulty of reputation building, I'd expect some kind of centralization to occur here, just like Bitcoin exchanges today are, to some degree, centralized entities.

If you have a direct payment channel to E and E is also the escrow service of that payment channel, then E has two out of three private keys, which is sufficient to unlock any locked micro-transaction funds, so this situation is really sensitive to abuse by E. Note that this situation can also occur by a sybil attack, or by conspiracy between E and either A or B.

In the event that E abuses such a situation, though, it is still possible for the victim to publish cryptographic proof of the abuse. This would destroy E's reputation, so hopefully that will deter E from performing the abuse in the first place. This would only work though, if E's reputation is really worth anything, e.g. if E is a large business that profits from having lots of other customers.


Why would W's distribution need to be determined by a live data feed? Maybe you need to clarify your concept. In my concept, the non-final versions of W are only exchanged between neighbors, and in case of a conflict, to the escrow service, and in case of escrow service misbehavior, to the public. The final version of W is sent to the public (on the block chain). Publishing to other parties wouldn't hurt the integrity of the system, but it would be a loss of privacy.

I do think though that, for the report of proof-of-(abuse by an escrow service), there should be a P2P exchange between users of that escrow service. It is essential, of course, that there should be no way for the escrow service itself to control this P2P network: it has a large interest in censoring such a network.

I'm now implementing this in Amiko Pay. In the implementation, I made a minor change to the concept:

To reduce coin fragmentation, I've put the funds of all locked transactions of a channel into a single output. In case of disagreement, the escrow service calculates how much of the locked amount is sent to which party. To enable the escrow service to do this, the amount of BTC in a microtransaction will be added to its TCD.

The down-side is that only a single escrow service can be used at a time for a micro-transaction channel; this is not considered to be a problem, since both sides have to agree to the choice of escrow service anyway. It is possible to switch to a different escrow service without closing the channel, if both sides agree to the choice of the new escrow service. In case they can not come to an agreement, then it's just a matter of settling the on-going transactions and closing the channel.

Note that this reduction of coin fragmentation is no longer possible once real Ligthning-style HTLCs are used. Also note that this coin fragmentation only really occurs in case of non-cooperating (mis-behaving) neighbors: cooperating neighbors will always settle their micro-transactions.

My major concern is that coin fragmentation leads to a large withdraw transaction and high fees; if the fee becomes high w.r.t. micro-transaction amounts, this could change the incentive-structure.

The most significant thing that happened was that the Lightning Network concept was published, which is very similar to the original Amiko Pay concept. I examined the Lightning Network and came to the conclusion that it achieves the same things as Amiko Pay, while it requires much smaller changes to Bitcoin to become completely "trust-free". So, my plan is now to implement the Lightning Network concept in Amiko Pay.

Since the Lightning Network still requires some Bitcoin features that currently don't exist yet, I went looking for ways to deal with this absence. Read this thread for a description of the design I made for this.

You are right that it is a very distinguishing feature of the Lightning Network to allow payments between people who do not share a direct link. The Lightning Network uses HTLCs as building blocks in a larger system that accomplishes this.

The answer to your question is: yes!

These emulated HTLCs can be used as building blocks in exactly the same way as Lightning's HTLCs. In fact, they can be mixed: in one and the same network, some links might use emulated HTLCs and others might use true Lightning HTLCs, and payments can be freely routed between them. In a way, this is similar to how some links in the Internet use Ethernet, some use PPP or something else, and Internet Protocol traffic can be freely routed between them.

In essence, all a HTLC has to accomplish is that, if a payment token is given before a time-out passes, funds go to one side, and if the time-out passes before the token is given, funds go to the other side. Then, HTLCs on different links can be chained together, by requiring the same payment token on each link, and using an increasing time-out when going from payee-side to payer-side. This is described in the Lightning paper.

Requirements for inter-operability between links seem to be very low. The same hashing algorithm has to be used on all links. The same address space has to be used for routing, but even cross-address-space routing might be enabled by gateway services(*). In fact, now that I think of it, you don't even need to be on the same block chain: this might enable trust-free, de-centralized exchange between different crypto-currencies. Trust-free exchange with fiat currencies is more difficult (read: impossible), since fiat currency is not bound to cryptographic rules. However, "colored coins" which represent fiat currency can be moved in this way, and even exchanged against non-colored coins. I already described that in a different thread.

(*) This is similar to how you can use IPv6 even when your ISP doesn't provide it, by tunneling your IPv6 traffic over IPv4 to a gateway service that connects you to the IPv6 Internet. Analogies with the Internet seem to be everywhere.

Emulation of Hash-Time-Locked Contracts of the Lightning network by a trusted, but publically auditable escrow service
http://cornwarecjp.github.io/amiko-pay/doc/lightning_emulation.pdf

I'm thinking of implementing this in Amiko Pay.

Looks suspiciously similar to my Amiko Pay, except that the commit conditions are different. I'll have to dive into the details of their proposal to figure out whether this is an improvement or not. Their proposal also seems to require additional Bitcoin scripting functions, but possibly less intrusive ones than my proposal. I'm also interested in to what degree a useful implementation of this can be made without those scripting extensions (presumably that would be less "trust-free").

And of course: I'll think about whether their commit conditions leave any vulnerabilities / abuse modes. It can be really hard to get these things 100% right.

Thanks. So I guess this means that, at the moment, sequence numbers have no meaning, except that the special value 0xffffffff makes transactions final. For the rest, it makes no difference what sequence numbers you use.

For most people, this will be the answer:


If you did compile your own software, then you can run "make check" in the source tree to see if you're affected. If all tests pass, you're not affected. You might want to check again after you update your system's OpenSSL.

Those who compile their own software can fix their software by applying a patch. The required changes are available on Github; e.g. here for the 0.9 branch.

I created a version of the 0.9 sources that's nearly identical to the official 0.9.3 source code release for Linux, but with the fix applied:
https://github.com/cornwarecjp/bitcoin/tree/b146f97935d6c17927406ea549409d232eb7ce3c

I wouldn't recommend doing development on that branch(*), but since it's nearly identical to the official release source code, it should be OK for compiling your own Bitcoin binary. Check for yourself with a diff tool what the differences are with the 0.9.3 sources and make sure you agree. In Linux desktops, you can e.g. use the "Meld" program for this, and use it to compare directories.

(*) The reason being that it's become quite different from development branches, which might make it more difficult to merge things.

...and after applying the patch, Bitcoin passes its test again.   Good work!

I now confirmed this, by first successfully running the Bitcoin 0.9.3 test suite, then upgrading OpenSSL (it still says 1.0.1e), and then getting a failure from the test suite:
http://www.ultimatestunts.nl/bitcoin/bitcoin_openssl_unittest_result.txt

Do you mean unsafe for a miner who considers accepting a transaction into his block, or unsafe for a recipient of one of the versions of a transaction (who might receive less or nothing at all in an update)? The second one really depends on the greater "scheme" in which transaction replacement is used; I haven't given the first one much thought yet.

That's just like double spending, right? So it isn't unique to transaction replacement at all: even without transaction replacement, attackers can "flood the network"(*) with double-spends, until one of them gets accepted into a block. The only difference created by transaction replacement is that the attacker, like a legitimate user, can indicate which of the "double-spends" should be chosen over the other ones. This can also be indicated without transaction replacement, in a more implicit, economical way, by increasing the transaction fee. If there is any difference at all, the explicit way with sequence numbers should be more reliable.

Again, the consequences for a recipient of one of the versions of a transaction completely depend on the greater "scheme" in which transaction replacement is used. The "scheme" I'm thinking of is a microtransaction channel; in that case, one of the sides of the channel can not attack the other side of the channel by flooding the network with transaction updates: the signatures of both sides are needed to create a valid transaction update.

But, please, PLEASE answer my question in post #12. It's much more urgent to me than this discussion about transaction replacement.

(*) I'm not exactly sure what you mean with that phrase. Do you mean it could be a DoS attack on the network?

Debian published this update:
https://www.debian.org/security/2015/dsa-3125

For Wheezy, the version number is still 1.0.1e. However, the description says it solves CVE-2014-8275, which is exactly the change that should trigger the Bitcoin problem.

So, on Debian Wheezy, the latest patched 1.0.1e can also cause problems? I guess I should first apply the Bitcoin patch, before applying this OpenSSL upgrade...

Besides the discussion about what Bitcoin's behavior should be, I'm also interested in what the exact behavior of Bitcoin is, right now.

The way I understand it, the current behavior of non-modified Bitcoin Core is (assuming all else is correct about a transaction):

• any sequence number is 0xffffffff -> see "transaction OK"
• all sequence numbers are not 0xffffffff and nLockTime is expired -> see "transaction OK"
• all sequence numbers are not 0xffffffff and nLockTime is not expired -> see "transaction NOT OK"
• "transaction OK" -> transaction will be transferred to other nodes and included into blocks. Will mine on top of blocks that include such transactions.
• "transaction NOT OK" -> transaction will NOT be transferred to other nodes and NOT included into blocks. Will NOT mine on top of blocks that include such transactions.

Is this correct?

It is useful for making microtransaction channels bidirectional. This is also possible without transaction replacement, but at some cost:
https://bitcointalk.org/index.php?topic=814770

What do you mean with "can never be enforced"? If a replacement is broadcasted long before nLockTime expires, it is unlikely that any miner doesn't receive it. Once nLockTime expires, each honest miner will include the latest update into its block. Most non-honest miners will do this too, because their only interest in your transaction is to capture its transaction fee. Only non-honest miners who have a higher stake in your transaction might choose to ignore a transaction update.

So, as far as I can see, the only "enforcement" you need is against a certain (typically very small) percentage of miners which might have a special interest in cheating. This "enforcement" could be in the form of other miners who refuse to build upon a block with an outdated version of a transaction. I'm not sure this is a good idea though: you need to be absolutely certain it won't ever break consensus in the long term. IMHO, preventing a long-lived block chain fork is much more important than any feature provided by trying to make transaction replacement 100% secure.

I have reasons for reconsidering the license of my Amiko Pay software, and because of that, I stumbled upon the Affero General Public License. This free software license requires service providers (typically, the operators of websites) to publish the source code of the service they're running.

I want to ask you, members of the Bitcoin community, how you feel about using this kind of license for Bitcoin-related software. It has the potential to push an entire ecosystem to become open-source-only. This would have interesting consequences: for instance, if an exchange uses AGPL-licensed software, you can easily request their source code and start a competing exchange with exactly the same set of features. It would also, at least in theory, provide some transparency about e.g. how your data is handled in a web service.

The obvious downside would be that people / organizations who don't want to share their code have more difficulty entering the ecosystem; this could create a barrier for adoption. Maybe this is a trade-off between achieving fast adoption of the monetary freedom provided by Bitcoin, and achieving software freedom.

I believe the Bitcoin Core software has a MIT-style license, so it opts for maximum adoption of the software, at the cost that some adopters might keep their software closed-source. Is there consensus that this is the best model?