I realize that a block can be empty - with no transactions - but don't the node operators still have to download the full file sizes? I know a block always has these values:
Magic no (value always 0xD9B4BEF9) - 4 bytes
Blocksize - 4 bytes
Blockheader - 80 bytes
Transaction counter - 9 bytes
Meaning that an empty block would require, at a minimum, 97 bytes. However, if the block size is set to 1 MB allowing that to be filled with transaction data, the question is if the nodes end up having to download a block file size of 1 MB if empty, or 97 bytes if empty. Like, the difference between an image file that contains padded zeroes and one that's been trimmed. If they don't download more than they have to, that would mean they do scale down as needed. If not, it really should work that way ESPECIALLY if people are talking about increasing the block size.
The second part of the proposal though - having the maximum block size increase dynamically according to the capabilities of the network and to provide incentive to node operators to make the required upgrades to allow the block ceiling value to be increased dynamically, what about that?
I fully agree that a limitless size would cause problems, and there should be a reasonable ceiling value set, but I still don't see why that limit can't be set by rules rather than opinion. Instead of long delays in making needed network changes, it could happen the moment a majority of the node operators are ready to handle it. By locking out node operators who are not scaled up with the majority, and providing financial incentive to node operators for higher bandwidth usage, it would create a similar arms race to the ASIC miners to ensure that the network can expand rapidly without the need for arguments over every little increase in block size.