I realize that a block can be empty - with no transactions - but don't the node operators still have to download the full file sizes? I know a block always has these values:
Magic no (value always 0xD9B4BEF9) - 4 bytes
Blocksize - 4 bytes
Blockheader - 80 bytes
Transaction counter - 9 bytes
Meaning that an empty block would require, at a minimum, 97 bytes. However, if the block size is set to 1 MB allowing that to be filled with transaction data, the question is if the nodes end up having to download a block file size of 1 MB if empty, or 97 bytes if empty.
97 bytes.
There is no required block size.
There is a block size LIMIT. Meaning that blocks are not allowed to be LARGER THAN the limit. Blocks smaller than the limit are perfectly valid and happen all the time.
I still don't see why that limit can't be set by rules rather than opinion.
The limit IS set by rules.
Changing those rules currently requires a hard fork. Hard forks require an overwhelming majority of users to agree. Getting agreement requires swaying opinions.
Consensus is hard.
Instead of long delays in making needed network changes, it could happen the moment a majority of the node operators are ready to handle it.
And how would you count "a majority of the node operators"? There is no reliable way to know how many node operators there are, and it is cheap and easy to set up millions of programs all pretending to be nodes in order to "stuff the ballot box".
By locking out node operators who are not scaled up with the majority, and providing financial incentive to node operators for higher bandwidth usage, it would create a similar arms race to the ASIC miners to ensure that the network can expand rapidly without the need for arguments over every little increase in block size.
The problem with autoscaling is that there isn't a reliable metric that can be used to determine when the size should scale up.
The difficulty can be scaled according to the time between blocks. This is a reliable metric. An attacker can't change the time between blocks without actually completing the proof-of-work (in which case if they are able to do that, then the difficulty NEEDS to increase).