It's gonna really tough for me to give up on my Monero

Why would you give up? Selling and repurchasing at lower prices is giving up?
Or do you mean that you bought at higher prices and feel not good about selling at a loss?
P.S. the only argument I could make for why altcoins would not go down while
BTC and gold are, is that it might be more difficult to short them? I don't know.