To incentivise inclusion of tx's in blocks, I propose the following (I work in hardware R&D, so I have only passing familiarity with the protocol, and so I may not know what I'm talking about!)

Solution: Miners switch to the longest chain (no change to protocol here), which leaves the smallest pool of unconfirmed transaction (addition to protocol - soft fork).

From my understanding, miners currently have an incentive to limit transactions because, in the rare event of 2 blocks being solved nearly simultaneously, the smaller block with propagate through the network more quickly. Larger blocks have a smaller risk associated with them.

So how do you incentivise including as many transactions as possible? My thinking here was that you cannot have miners give preferential treatment to the block with the largest number of transactions. This is because a deceptive miner could generate a large number of spam transactions without broadcasting them. Any block they solve could be padded with their secret tx's, artificially giving their blocks preference on the network.

Instead, if miners were to see 2 blocks solved nearly simultaneously, they could switch to whichever chain gave the smallest unconfirmed tx pool. Perhaps some details to prevent miners from toggling chains may be in place. For example, preference=mod(tx's,100) so that miners don't unsessisarly jump between preferred chains.

From my understanding, this would not require a hard fork, because no miners would not need to necessarily work on the preferred chain as I've described above. By participating in whichever chain they receive first, they simply risk other miners preferentially working on a different chain during the race duration.

---
P.s. I've been using bitcoin since beginning of 2013, but haven't gotten into the forums. Just wanted to thank everyone for such an awesome experiment! I know physics more than programming. Wish I could help more!