Why would you go to a banker to buy physical gold? Thats what bullion dealers are for, hand over the cash and walk out with the metal.
You said the right words. When Gold is the money, those banks just want 20% profit every year by saying some words like "wealth management etc.."