Certantly the UTXO size is a major concern for the viability of the system, since it sets a lower bound on the resource requirements (amount of online storage) for a full node... but it is not held in memory and has no risk of running hosts out of ram as you claim.
One of the costs of UXTO is it must be downloaded by every ephemeral miner that wants to be a full node (well I guess I am conflating UXTO and the entire block chain). This has been the source of some complaints against CN coins with large block chains. Now granted that given my points recently, solo mining is really pointless especially for Bitcoin where you need to amortize an ASIC purchase. So this is a non-issue for Bitcoin (but only because Bitcoin is already centralized) whereas it is for CPU mine-able coins such as the newer CN coins.
However, I must take issue with your assertion that unbounded UXTO could never cause a full node to run out of RAM. Afaics it is possible to envision a level of swapping that the system can not keep up with the transaction rate. That is however perhaps irrelevant if for example solo mining is not viable due to variance for small hashrate and where bandwidth (or latency) being the predominant bound for miners with less resources (or who wish to remain ephemeral for privacy reasons and any other reasons).