I think it depends where you are from.
In the US I believe the IRS follows you everywhere.
In Australia if I am correct it is in simple terms based on residency so if I live in Aus but have a wallet setup in say Germany and I use a German exchange, I don't think it matters. But don't listen to me :|
As far as I know only the US charges you taxes for being a citizen no matter where you are around the world, the rest of the countries charge you taxes based on residence so if a person that is not an US citizen has a lot of bitcoin they could just move to a friendly jurisdiction when it comes to taxes or bitcoin and then sell a part of his coins there and save himself a lot of money that way, but for the average holder this is simply not an option as they do not have as much bitcoin to justify this move.