https://github.com/bitcoin/bitcoin/blob/master/src/script/interpreter.cpp#L431

This EvalScript function is basically the script VM.

I have a question for anybody familiar with the Bitcoin codebase. I'm interested in how spent/unspent outputs are tracked to prevent double spends, specifically I just want to know what data structures are used for checking if a output is spent or not (i.e. to prevent double spends). I know this could be done with a hash table or bloom filter.

I took a quick look through the codebase, I would have expected it to be https://github.com/bitcoin/bitcoin/blob/master/src/txdb.h#L59 although this HaveCoin seems to check the existence of the coin - not if it's spent or not.

It would take me a long time to read through the code and figure this out so I'm hoping somebody who already knows the answer can help me.

Many thanks!

Very good read! I'll just leave it here.

https://computersciencealgorithms.wordpress.com/2020/11/22/exploiting-arbitrage-opportunities-in-betting-markets/

Makes sense. I think some block explorers keep them, but probably just for data analysis purposes.

Great answer.

I think the next step in blockchains is distributing load - storage, CPU, network, etc. in a fair and trustless way, while maintaining the distribution in power. Blockchains such as bitcoin are vastly over-redundant.

Speaking as somebody who has tinkered with the C++ codebase a few times, I find a good way to learn it is to first look in the test directory, and run some of the tests individually, try them with different inputs, etc.

All the tests for most of the core protocol stuff are in here - scripts, base58, transaction, etc. So you can relate the core data structures back to all the books you've read on the topic.

It's not the easiest code base to understand, I find it's helpful to use an IDE with a bookmarking facility so you can "map out" the important parts.

Good luck!

So no, ok cool. Thanks!

Suppose I'm writing an app that uses the Bitcoin blockchain to store small amounts of data - eg. hashes, I write a tx to a block and later that block becomes orphaned. The tx probably will get confirmed in the forked chain, either at the same height or later.

But if not, is it considered an abuse of the protocol to refer to the orphan block in my app - for example storing a reference to the orphan block in a database so the tx can be retrieved later? Are there any guarantees about the availability of orphan blocks, in bitcoind or other implementations, or are they prune-able ?

How long does Utreexo take to sync the blockchain? It says in the medium article "A full node in a few kilobytes that syncs as fast as an ssd on a hdd". I guess it does everything in-memory and doesn't write the B/C to disk. There's still a CPU / Network bottleneck here.

Anyway, interesting project.

awesome. thanks!

Many thanks for the detailed reply.

You just made me realise something - actually the use case I had in mind does require accountability - in fact the N keys must be known and provable at time of locking and the M signers must be known or somehow deducible from the spend at time of spending. So I guess the super-efficient threshold signatures are out of the question.

The tree of all possible signatures seems like it *could* work for what I want to do, although the tree could get very big too.

Would you be able to point me to any resources (code, medium, etc) which explains these trees with more clarity?

A problem with this approach is that the block will contain a coinbase tx which includes a recipient address.

Everybody wants to be the recipient of the block reward which will entail changing the recipient address, which will change the hash. Also, as achow101 pointed out, you can not "aim at" a particular hash. The amount of currently done work on a block could give miners a clue as to what other miners are mining, but I don't thing they would find this information particularly useful.

I'm hoping somebody who has a better understanding of Schnorr signatures than me could answer this question. I have a basic understanding of EC math but have not studied Schnorr in great detail.

As I understand, Schnorr signatures as implemented in Taproot allow a script to be created which requires a m-of-n threshold signature in order to be spent.

My question is: is it possible for a completely non-interactive locking of funds? i.e. scriptPubKey. Of course this is is possible with traditional scripted threshold signatures, but as I understand Schnorr signatures requires the signatories to choose random numbers to add to their public keys before a master public key is created. Without these random numbers there is an attack that can be performed by adding the public keys together. Several questions arise from this:

1. Is it possible to choose these random numbers in a pseudo-random, deterministic way, yet still be safe?
2. Do the signatories need to remember the random numbers in order to create the Schnorr signatures at time of unlocking?

Apologies for my really poor understanding of Schnorr. I'm in the process of learning about it.

I appreciate if anybody could help me with this question.

True. and the fees got mighty high at those times too, for example there was a period from Apr '17 to May '17 when the blocks were nearly all full. During this period the mempool size peaked at about 150Mb, but in the December spike it doubled to 300Mb.

Nope. There's a non linear relationship between $ fee and BTC price. The fee in satoshis increased too.

I think a lot of investors around this time did not understand the technical limitations, and this was the first bubble where the 1Mb limit actually came into play. And the $30 fee would not have remained $30. It would have gone up had the bubble kept going. Who knows how high?

I think there was a large contingent of smart money who saw the crash coming and sold once the fees started going mental, but it's hard to argue that there was a strong correlation between hitting the 1Mb limit being hit... I mean just look here, take a look at "Pending transaction fee in BTC". It's clear to see that the fees (in BTC, not $), went mental just after Dec 16th.

It's hard to argue that the fees didn't play a role, and the fees are dependant on the block size.

Nobody disagrees with you that it was a speculative bubble. I think even my dog knows it was a speculative bubble. But there's a lot of data to suggest the pin which burst that speculative bubble was the fact that once the 1Mb limit was hit, the fees just went fucking crazy and people were like "fuck this, I'm out". If the 1Mb limit was not there, the bubble could have gone a lot higher. 30k, 50k, maybe even 100k.

Take a look at these charts:

Block size
GDAX BTC/USD market
Media transaction fee
Pending Tx fee chart (denominated in sats/byte

If you look at the block size chart, from October to November 2017 it was getting close to the 1Mb limit, then for virtually all of November it was consistently at ~960k (and never below 940k) and in December it was almost always above 960k. It was running very hot at this stage. In late November the median fees were ~$5 (0.0007). At this time BTC was trading at around $5k (VERY conservative estimate), and the fees actually weren't too bad, they were a median of around $3. so 3/5k = 0.0006 BTC / Tx. In early December there was a massive pop in the price where it rose from about $10k to about $17k, and there's a corresponding jump in dollar fees from $3 to about $15 = 0.0009 BTC / Tx. 50% increase in fees.

But then look what happens. On 16 Dec (a Saturday) the price shoots up to $19,500 and during the following week (close to Christmas), the money starts rolling in and the blockchain gets very busy, and the median price per tx shoots up to $33 (Dec 22), by this time the price had started to crash a bit, but nobody knows it's already peaked. So on Dec 22 thee price was down to around $14k, but the median tx price was $33, $33/$14k = 0.002 BTC / Tx =a 2X increase in dollar terms and more like 2.2X BTC terms.

So the fees were basically doubling every few days - not just in dollar terms due to the rising price of BTC. But in satoshis too. It's pretty damning evidence and hard to argue with. The 1Mb cap was hit.

I recently made a post about an idea I had for standardising OP_RETURN outputs such that data may more easily be stored on the blockchain, and I received a lot of resistance and opposition, For me this makes no sense and data storage is one of the killer apps of blockchain.

I get that BTC core people want to protect their precious 1Mb of block space and use it as efficiently as possible, and right now, that probably is the somewhat right approach (like, we don't want the price to tank, right?) I totally get why they don't want tons of data going on to the BTC blockchain and inflating transaction fees, possibly causing the price to tank as it did in the 2018 crash. I strongly believe that this crash was mainly due to the fees, as it coincided perfectly with the upper TPS limit.

Interested in hearing people's take on this.

I don't agree that encryption will inevitably be broken. That's a straw-man argument. If you're really worried about encryption, nobody said you had to use a standard encryption algorithm like AES-256. You could use a 512-bit or 1024 bit AES if you like. If you're worried about quantum attacks you can use a quantum-resistant encryption scheme. Heck, you can use 10 different quantum algorithms and 10 classical algorithms strung together.

I never said anything about encrypting anyway - the data could be non-sensitive data. It could be hashes or timestamps. If it is encrypted data, then it's really no more risky than storing it in the cloud. In fact it's a trade-off. If you store your data in the cloud you run the risk of them losing your data or confiscating it. When it comes to encryption you have 2 choices with the cloud: they manage the keys, or you manage the keys. Of course you can do both and double-encrypt your data.

The bottom line is: you need to take care of your own keys. If they get compromised, either through a physical attack or cyber attack, the attacker will be able to access your data whether you store it on a private cloud or on a public blockchain. So the issue comes down to key management. There are several ways to do this, most notably key distribution so there are multiple keys in different physical locations.

The bottom line is: If you are a law firm, and you have sensitive data, any time you need to access that data, you need to grab the encrypted data from somewhere, decrypt it, then view it, or do whatever you need to do. Whether that somewhere is a public blockchain or private cloud is irrelevant, so long as you're using a decent encryption scheme. It's just transferring bytes which are absolutely useless to anybody not in possession of the keys and with knowledge of the encryption scheme.

Typically I would expect people to store small text files, not jpegs, because they cost too much. If somebody wants to put a small image of their new born baby or a recently lost loved one, I would expect it. But such an image could cost $10 or more so market dynamics will decide


There already is illegal images on the bitcoin BTC blockchain, so you're hosting them on your node. I don't expect this to become the de-facto method by which sickos distribute their sick images. It makes no sense economically and the blockchain makes it easy for them to get caught, so I don't expect this to be a problem.


Not wanted by you, maybe. Do you speak for the entire community? I'm sure many people would welcome a way to store small, critical pieces of data, safe in the knowledge that the data can never be lost and they can access it any time in the future.

This is not just about private individuals storing their favourite cat pictures though, you need to think bigger. How about law firms storing important legal information? Think bigger.


I don't give a flying fuck


Bitcoin SV are very bullish on this idea and are already doing it.


Usenet is 90's tech. Torrents are 00's tech and also outdated. Torrents are great for sharing transient data but if you want persistence torrents are useless. Your response is not just condecending, but shows up your mental laziness.

 This is not the use case I'm discussing.

Ok that's my last post in this thread. Mod, can you please lock/delete this thread ? I'm clearly wasting my time.

Data ~= Files 

A file is nothing but some data with an associated key. What I'm talking about is storing data, with a key so it is easily retrieved. Files are traditionally identified by an extension such as ".jpeg" but this is quite wasteful so I propose replacing it with a number (maybe 3 or 4 byte or a VarInt). So you have:

1. Raw data
2. Key (filename)
3. Data type (1=jpg, 2=png, ...)

FYI, Bitcoin SV is already doing something like this, and they already have a system devised for splitting up large files into several OP_RETURN transactions, but I'm not sure if they have a keying mechanism. That is the icing on the cake.

here is the official MS block post on it.

It's built on Bitcoin core and IPFS and MongoDB. I think Bitcoin is mainly used as a timestamping mechanism, but there you go...