I'm still lacking answers on the whole "what if I decide to get a large crew together to violently take your shit" argument I presented earlier, as well.
That's why we have nation-states today. The great 19th century innovation was total war. Note that takings may be motivated by ideology as well as by colonialism. Mutually assured destruction was effective at preventing a hot war during the late 20th century standoff between eastern and western blocs. Studies of deterrence factors for crime indicate that severity of punishment is somewhat effective, but the most effective factors were promptness and assuredness of punishment. Prompt and assured proportionate retaliation is the most effective deterrence to aggression. This can be decentralized as well. Switzerland has done this quite effectively. Since every household was well-armed, even after fully encircling Switzerland, the Germans sought rapprochement rather than invading.
Switzerland in the WWII era is informative. When dealing with a hostile threat, it's not necessary to possess the ability to defeat the aggressor. It is only necessary to present a less tempting target than the alternatives. Empires always crumble when the costs of administering colonies exceed the benefits of exploiting them.