The recent discussion about some politicians wanting to introduce KYC/AML requirements for miners (which I of course do hope will never come, in no country of the world) brought me to think about the censorship problem, i.e. the dangers of miners blacklisting certain UTXOs or addresses.
Currently, Bitcoin's censorship resistance is based purely on incentives. A miner can include or censor the transactions he wants to, but if he doesn't chose those with most fees included, he will make less profit. And if a transaction gets rejected but pays significantly more fees than the current lower bound to be included, it is likely that another miner will include it in one of the next blocks.
Now: What if e.g. 99% of miners blacklist an address or UTXO, would the current system be enough? Would there be a way to improve that, introducing new protocol rules?
I have basic understanding of computing and blockchain tech but not advanced enough to know if an improvement is principally impossible or if there only hasn't been enough research on this.
If there was research, even the goal it was deemed impossible, I would be grateful for links, possible BIPs, mailinglist discussions etc.
I (as a layman) could imagine, for example, the following mechanism: Miners could publicly first sign all transactions to acknowledge they have received them. Three new rules are introduced for tx inclusions in blocks:
1) They can only include transactions which they have publicly signed before.
2) They cannot include any transaction which has a lower fee than another transaction they didn't include but have previously signed.
3) Any node ("challenger") can challenge a recent block proving that the miner hasn't followed rule 1 or 2, and if he can prove it, then the "challenger" node gets the block reward and fees (this would need major protocol changes, but should be possible as similar mechanisms exist in "slashing" PoS protocols).
This is in the end also a "financial incentive" to not censor, but it should be much stronger than the current mechanism as an entire block reward + fee would be in danger for the censoring miner.
The problem is, obviously, "where and how do the miners exactly publicly sign the transactions"? There would have to be a "global state" of all these signatures.
If they have to sign the complete transaction data on-chain (i.e. as an "attachment" to an earlier block, which later could be pruned), then they could censor transactions in this step based on blacklisted UTXOs, like they would when they decide which transaction they include in a block. Probably nothing would have been won, although I could imagine situations where a "cartel" censoring transactions could have to be bigger with such an approach if two different miners have to intervene in a "approval" of a transaction.
Where my doubt is if there is a way to make them sign only the TXID or incomplete transaction data without the identification of UTXOs, without possible "challengers" being able to game the system, never transmitting the complete data to the miners. Is this perhaps possible with zero-knowledge approaches? Or could there be a second step, i.e. once the challenger has proven a miner has violated the rules, the miner gets some kind of second chance (as then the transaction data would be publicly known due to the challenger)?