If you have 1 MB block size limit and each input is approximately 250 bytes, you only have at most 4000 inputs per block.
Well, the size limit (at least the 1MB version) is not likely to last indefinitely.
My point is that we can associate performance requirements with block size rather than with UTXO set size.
If you think about it, it makes more sense: block size limit is the part of the protocol. It is something which can be controlled, and is being controlled now. But UTXO set size might be as high as 21*10^14, and controlling it is rather problematic and non-trivial.
So it is possible to design software in such a way that block size limit will dictate hardware requirements for full nodes. If there is a need to increase throughput, block size limit will be increased and hardware requirements will be adjusted accordingly.
So you can just tell full nodes maintainers which hardware they need to keep up with blockchain growth. It will give them peace of mind. Isn't that nice?
However, it does reduce the overall performance of the system.
If it is harder to spend "old" transaction outputs, then that weakens the ability of the system to act as a long term store of value.
You can prioritize them by value. E.g. if size of 1 UTXO in memory is 100 bytes, storing 21 million entries requires only 2.1 GB, and thus you can easily guarantee that all UTXOs which have at least 1 Bitcoin in them will never be flushed away.