In the short term...to high a limit and we get increased risk and spam attack risks, too low and we get transactions queued. Both are very bad. Whether you think one is worse than another is a matter of perspective.
Well I think this is a common misunderstanding. The block limit never had and never will prevent spam. When the 1MB limit was put in place the average block size was about 35KB. So even after it was in place a miner could have made a block 3000% of the average block size. The rules for standard transactions, transaction priority, dust threshold, and minimum fees is what prevents spam. The block limit only provides an upper bound on how much damage a malicious user could do. By your definitions the 1MB limit was "too high" as it never did anything to prevent spam. In fact it failed utterly horribly at that job (well failed as much as something failing to prevent an event it was not designed to prevent). Prior to the dust threshold being created users (one service in particularly) horribly wasted the most critical resource of the network. No contrary to popular opinion that isn't block space, it is the UTXO set. The UTXO set is critical because blocks are not used to validate new blocks or transactions. Blocks are used to update the UTXO and the UTXO is used to validate all new transactions. Under normal use the UTXO set grows slower than the overall blockchain. Satoshi Dice sent low (as low as 1 satoshi) outputs to notify users of losses. Those outputs will probably never be spent but the network can't "write them off". They will remain perpetually bloating the UTXO set. Even if the block limit was ultra conservative (say 100K instead of 1000KB) it wouldn't have prevented this poor use of critical resources.
So what did the block limit do. It provided an upper limit on the damage that one or more malicious entities could do to the overall network. If 100% of the network conspired it still put an upper bound on the blockchain growth of 30x the average block size. If only 10% of the network conspired then it limited the bloat to no more than 3x the average block size. A single malicious entity creating the occasional max block would have even less effect in the long run. That was the purpose of the block limit. The block limit is like fire insurance for your house. It limits the damage in the event your house is destroyed but it would not be accurate to say that fire insurance prevents fires anymore than the block limit prevents spam
With all due respect, contrast this limit (or any limit) with unlimited.
It does indeed prevent spam attacks. Do not confuse the scope of the threat with the existence of it.
We are looking at arbitrary amounts of risk of threat to accept, and the proposal is for 16x the current risk and x16000 over 20 years.
That may be a reasonable number, or it may not be. We can't know from where we are today.