If you are a block creator, then you have an incentive to create the biggest possible block. Why? Because then, you can pick some simple, deterministic algorithm, and generate terabytes of always valid transactions on-the-fly, then let your miners work on that header, and send mined block to other nodes. Then, you can start producing some next block, on top of what you created, while other block creators will try to validate what you submitted.
That doesn't make sense, unless such miner/pool have goal to bloat Bitcoin blockchain. With such long verification time, it would hinder propagation where other miner could beat you since their block is far faster to be verified and propagated even though they mined it few seconds to minutes after you did.
In order for Bitcoin to succeed, blocks must be full.
If blocks are never full, then there is never a reason for a transaction to pay more 1 satoshi in fees. As the subsidy is reduced, fees become more important. So at some point, full blocks will be necessary in order to ensure that the revenue is high enough to discourage a 51% attack.
Changes that attempt to prevent full blocks would directly impact the security of Bitcoin.
Can't we have flexible block size? For example, each block confirming 4% of total transactions (numbers are only for example). There will always be some unconfirmed transactions left because block only takes percentage. If 100K transactions are unconfirmed at the moment, 4K will get confirmed in next block, if the demand rises significantly and we have 300K unconfirmed transactions, then 12K will get confirmed in next block. By doing this, the less total transactions we have, the less transactions will be picked up which won't let transaction fees to fall down but at the same time, if demand significantly rises, it won't let big transaction volume to clog the mempool.
Blocks can be always full if block size is flexible and changes for each new block discovery, according to the volume that 3% of transactions create at moment. I don't know how technically possible is that but I don't think that sounds dumb in theory.
Each node have unique mempool or even disable mempool, so your idea can't be achieved since each node would set different maximum block size size limit.