Simple. Something doesnt necessarly need to crash if it stops being used a currency, if you can still sell it at high prices. People will pay you what they think gold's value is. Supply and demand thats all.
But today, the supply of gold is higher and the demand is lower. Yet even adjusted for inflation, gold is worth more today than it was 100 years ago.
So basically, ever since it became legal to once again own physical gold, people who understood a little bit about economics and inflation (in the 1970s we had stagflation, an even worse condition of a stagnating economy combined with inflation) bought gold as a hedge against the decreasing value of the dollar. Looking back through recent history, you will see that when crisis hits, demand for gold goes up.
So the demand from this small percentage of society was enough to make up for the bakers, farmers, and teachers who no longer demanded gold as payment for their goods and services?