When the Spanish conquistadors invaded Mexico and Central and South America, they brought home a great quantity of gold. The amount of gold they brought back home was enough to cause inflation back home.
Yes, if enough gold were found that made it as rare as dirt, it would lose the value it now has, in the same way that fiat currencies are losing their values.