There are multiple middlemen scenarios that can be imposed, if desired, and others which are less transparent. The most obvious being VAT. Platinum, Silver, etc already have VAT in multiple countries. So you can't (legally) buy or sell, without paying VAT. If the VAT, is, say around 20-25%, for every 4-5 times an ounce is transacted, the state has got back the entire value of the ounce in taxes. The same could be done in Gold with a few law changes.

There is also heavy mining centralization as governments practically own the ground beneath the surface. In most countries, if not all, you own the surface level of your house, but you don't really own the air above it, or the ground below it - these are rights reserved by the government. This gives the existing gold holders <1% of the gold that will ever be mined, and the states the 99%+ of the gold that will be mined in the future. This, in turn, allows for significant dilution of value (in fiat terms) of existing gold holders, if they can't "participate" in the "stock split" or "stock offer" - as quantities grow and grow but governments control the output.

I'm not saying this is happening, but there are various possibilities that can happen. Including the elimination of cash - which automatically means that all transactions will be monitored by big brother, including all buying and selling of gold/silver for e-fiat, unless the transactions are barter-like (which is ok but slightly primitive and suitable only for certain civilization breakdown scenarios).