Bitcoin isn't just gold, it's gold^2. Finite supply that appreciates over time. It's actually a diminishing supply once you take into account the lost wallets.
The best answer to this question is very simple. I will feel confident if I have gold in my hands, because Bitcoin is a virtual currency and, with small changes in policy regarding crypto currency, the user can lose everything.