Libertarianism is a model, an over-simplification.  It assumes that one can tell the difference between a competent, consenting adult and someone who is not a competent, consenting adult.  In the real world, there is no such abrupt transition.  This means, for example, that there can be no coherent libertarian theory of children's rights.  This doesn't mean that libertarianism is useless; it's still a good model for voluntary transactions between competent adults.  It just means that if you try to apply libertarianism outside its natural domain, the model breaks down.