why would a scriptpubkey need to be unlimited in size for what is supposed to be a "payment system"?
Only Satoshi can say the exact reason. We can speculate. Logic suggests the reason is for scalability and forward compatibility. Putting restrictions on output scripts as part of the consensus rules could introduce complications for future work without breaking backward compatibility.
you have to define a strict format for a transaction call it a transaction template and anything that doesn't matchup to that template gets rejected.
We have something like that called standard rules.
bitcoin really made a mess of things with its utxo model and using SCRIPT language and all these different address types.
Not at all. The smart contracts (the script) is the way that Bitcoin can work decentrally and the alternative ways (like account based instead of coin based) has its own complications and disadvantages as well.
used only public keys, and nothing else. There was no OP_CHECKSIG or anything like that.
Actually P2PK has a OP_CHECKSIG at the end, without it anybody could spend those outputs! Remember that every time you send your coins you are actually locking them up in a smart contract that can only be unlocked if the correct unlocking script is provided and executed automatically by the machine using the Bitcoin script language. So the "machine" has to be told to "check [the] sig[nature]" otherwise it won't do it therefore anybody could spend an output script with pubkey and without OP_CHECKSIG.
BTW even though initial outputs were all using P2PK pretty much all other OP codes existed and P2PKH outputs were also possible to create.