This is really matter of perspective. It's just harder to confiscate then other assets. There are ways for governments to strong arm your hand and seize stuff and they have done that in the past. For example US seized all citizens' gold in 1930s.

You can always decide to not give that privatekey information. But as buried gold, someone will be asking in the future how did you aquire it in the first place because ATM or FATM or whatever.