Even if they know which address belongs to you, as long as you don't need paper money, they can't do anything. They can't take it away, seize or steal.
Depends on who "they" are. Thugs can take your bitcoins at gunpoint. Government can do essentially the same thing, disguised as "taxes".