I thought that you'd use RSA or another
No I was assuming that we could do asymmetric encryption directly with the EC keypairs just like with RSA but it turns out that might not be true. I suppose we could use RSA but then the stealth addresses would need to be even longer than they already are, and they're already much longer than a normal Bitcoin address, so making them even longer seems untenable. If we could use the EC keypairs for asymmetric encryption then we would avoid that problem because those long stealth addresses already contain the EC public key. We still need that in order to generate Q' so we can't just get rid of it.