You are correct to question whether the incentives for miners coincides with what is optimal for the rest of the network. As we see with empty blocks and non-validated blocks these can differ. However, your assumption that these do not coincide whatsoever is unsupported. And with Bitcoin it is all we have, barring some centralized committee making arbitrary decisions like we have today with the core devs.
Bitcoin has consensus rules. You may view that as being something decided by a central committee (in a sense that is true because the consensus rules are encoded in the software) or alternately, as something decided by the consensus of users who run the software (in deciding whether to use a modified version), but either way there is nothing improper about those consensus rules including a block size limit just as they include other limits, for example, the maximum allowable size of scripts, which op codes can be used at all, whether coins can be created out of thin air, etc.
By your argument the consensus rules should be removed and miners can just include whatever they want. After all, miners won''t want to include anything that other miners wan't want to build on, so the whole thing is entirely self-enforcing. Maybe that sort of a coin would work, but it isn't Bitcoin at all. It's very close to a greenfield redesign.
Bitcoin by its nature includes consensus rules that are enforced by all users, not just miners.