Re: Are there any benchmark about Bitcoin full node client resource usage?
Your claim about transaction packages" not being relevant here is totally false and laughable
There is no such thing as 'transaction packages' in blocks from the perspective of validation.Packages are a concept used in selecting transactions to mine so that low-fee ancestor transactions will be selected in order to mine their high fee children. They don't exist elsewhere.
Quote
Quote
Quote
The verification of transactions runs in parallel with everything else. One thread loads transactions from the block into a queue of transactions that need to be validated, other threads pull transactions from the queue and validate them. When the main thread is done loading the queue, it too joins into the validation which has been in progress the whole time. There is nothing particularly fancy about this.
Absolutely false and misleading. The code does not "load" transactions to "queue of transactions" you are deliberately misrepresenting the code for some mysterious purpose that I don't understand.The transaction?
No! The input script of the transaction! And it is the point:
For queuing the transaction input scripts, the code checks for the existence of the input in the UTXO set. Doesn't it?
So, it is not about parallel transaction processing because other than their input scripts, transaction are processed sequentially.
Quote
Quote
In the real world, the real bitcoin core client, does NOT validate transactions in parallel,
It does. Since you're non-technical I don't expect you to read the code to check for yourself, but you can simply run reindex on a node with -assumevalid set to false. You'll see that once it gets up to 2014 or so that it is using all the cores.Comparatively your take from the code is, in the best sense, somehow "raw" because you think it is the transaction that is getting queued but it is not.
BTW, I don't get it, how "using all the core" is enough for justifying your idea.
Quote
Quote
because in bitcoin there is a possibility for transactions to be chained in a single block, forming a transaction package, hence you can't simply "dispatch" txns of a block between threads waiting for them to join, it is why block verification was implemented single thread.
Transactions in a block are required by the consensus rules to be topologically ordered. That means that all the ancestors of a transaction come first. There is no concept of a 'package' in a block.
When the validation is iterating through the block to load the validation queues it saves the new outputs created by each (as of yet unvalidated) transaction, so that they're available when dispatching the work off for any future transactions that consume them. They don't have to be validated before other transactions can consume them, because if there is any invalidity anywhere the whole block will be invalid.
So you can have e.g. an invalid TxA whos outputs are spent by valid TxB whos outputs are spent by valid txC, and its perfectly fine that the validation accepts TxB and TxC before later detecting that TxA is invalid. A's invalidity will trigger the rejection of the block.
Extracting the outputs for other transactions to use does require a linear pass through the transactions but it's fairly inexpensive and doesn't require any validation. It is required in any case because the block serialization can only be decoded in-order too (because you can't tell where transaction 2 begins until you've parsed transaction 1-- the format doesn't have explicit lengths).
Similarly, checking if a UTXO has already been consumed is also inherently somewhat sequential (e.g. consider when the 5th and 50th txn both spend the same input), but these checks are cheap. Often attempts to make processes like that more parallel just slow them down because of the synchronization overheads. That's why it's not possible to give much in the way of useful parallelism advice without testing.
And it is the problem, I suppose:
You get offended by any comment, no matter what! o
Few weeks ago someone in this forum told something about block verification in bitcoin being sequential because of "packaged transactions" (take it easy). I proposed an algorithm for this and the discussion went on in other directions ...
Later, I checked the code and noted the queuing thing and input scripts being subject to multi-threading instead of the whole transaction asking myself why not going after the whole txn? ...
Additionally, I noted the excessive use of locks and became concerned about their impact on parallelism.
Put these alongside a true story: being stuck with a week long synchronization nightmare just few days ago
Then I saw your comment on the other topic and felt it is appropriate to remind the partial nature of the way parallelism has been implemented rather than leaving people confused with a simple sentence: Transaction validation is multi-threaded!
My impression about the way PR2060 is implemented was and still is that there is a concern about the "packaged transactions" (come on, t is good term for the purpose) and I thought it is adequate to share.
But when it comes to Greg Maxwell, things get super sensitive. No?
You think every point is made for a purpose, a bad purpose probably, ...
It is not true. As an engineer and a researcher, I can't, and I won't blindly follow every single verdict BIG Maxwell issues, don't feel offended about it, because at the same time I have deep respect for you and other guys who have been serving on the board for a decade,
Honestly, this discussion didn't add anything to my knowledge about the code and bitcoin but I' sure many readers didn't know how exactly things get done in the validation process and found it useful and if you could keep it more informative rather than "challenging" it'd be even more useful.