That doesn't make sense to me. The formula wouldn't need to be arbitrary; it could be based on actual data.
But the formula remains arbitrary. You can't come up with an algorithm capable of measuring actual demand and actual supply, since these units are impossible to measure. So you can't really know how many security is demanded (remember, demand is subjective!), nor how such demand would compete for Earth's scarce resources. You need to be omniscient to know all that.
The difficulty target aims to make one block at every 10min. But why 10min? This is an arbitrary value. It may be too much sometimes, too little at other times. It's certainly not optimal. That said, it's not such a big deal, and trying to improve it would not be worth the risks.
Concerning mining remuneration, if we can go directly to spontaneous order - and that's the closest you'll ever get from "optimal" -, then why not? Why try to come up with arbitrary formulas? That would be "presumption of knowledge".
"The curious task of economics is to demonstrate to men how little they know about what they imagine they can design."
Politically, a cap is a less radical departure from the soft and hard block limit which people know about. Psychologically, it maintains a perceived need to add fees, and might price out SD-like flooding.
SD is not flooding anything. They're not attacking the network, Bitcoin users want to use their services.
Of all business, they're likely the one that has mostly contributed to miners via transaction fees.
It also prevents the chance that an unexpected monster block gets accepted and built on causing problems for some miners.
Miners have no interest in keeping a "monster block". And they can easily choose not to build on top of such block, unless it is N blocks deep already, what would likely get the monster block rejected by the network.