Quote from: goatpig on May 10, 2020, 01:15:37 PM

Has there been considerations for a stand alone key:val table for transaction short ids? I'm asking because I can't tell whether the benefit would offset the overhead. It "feels" like a boon for historical data but a mess of complexity the younger the blocks get.

Instead of 'short ids' one would probably use an output counter for that-- saves having to deal with collisions. It's cheap/simple to maintain the counter itself (there is one for the total txn count displayed in the updatetip log entries).

Even with no reuse of old spent ids (which would make maintaining the counter complicated), every input could be referenced using 31 bits right now. ... pretty big improvement over 264 bits currently used by bitcoin (or the 258 or whatever in the this new encoding).

I want to say when Pieter crunched the numbers doing that it was an additional 30% savings!

You could probably do a bit better by transmitting with transactions the highest counter in the transaction separately, and code all the other inputs as varint-like difference below that value. (Or if you want to be really elaborate: code the all the input indexes for the transaction differentially in descending order, plus log2(inputs!) bits to encode which goes to which.)

But you'd have to have an index on disk of counter->txid which would take up a lot of space, and take time to update. Sad

I think maintaining through reorgs it wouldn't be too complicated, because it would just ride along with the rest of the state. ... e.g. the txindex gets updated in that way.

When using this with loose txn (rather than in blocks) you'd also run into issues where the encodings weren't compatible between different peers who were on different near-tip forks. One way to handle that might be using a bit per input to indicate if the counter or full txid was used, and use full inputs on loose transactions for inputs with counter values too close to your current tip.

The additional savings is substantial. But because it would require fixed overhead even if you weren't using it (the counter->id index, which both potential senders and potential receivers would have to have), it's a little more difficult to reason about its prospects for deployment. The version blockstream put out has the advantage of being extremely self-contained in the codebase, and having no overhead (except the code) if you're not using it... so it's a realistic prospect to have all nodes adopt this and use it on a case by case basis.

I guess from my perspective, I proposed this sort of thing concretely in 2016 (and less concretely probably years before) and it took all this time for it to get implemented at all and there still isn't any real prospect of its use outside of satellite. If input reference compression had been a part of it .. it still probably wouldn't be done.