I think they can be compatible, but it depends on how you go about establishing each position.
If you take moral relativism to be the thesis that moral judgments all have the same truth-value (i.e. all are true, all are false, or they don't even have truth-values) or all have the same ethical value (no value is "better" than another), and you combine it with the empirical thesis that people will, in general, subscribe to and act according to many different moral values, then you pretty much end up with the current moral landscape.
Subscribing to libertarianism, even given this morally relativistic landscape, might still make sense though. You could go at it on consequentalist grounds. Since people will tend to act according to their own values (which are determined by socialization, upbringing, and whatnot), and since no values are ethically "better" than any other, it isn't reasonable to then argue that certain libertarian values are ethically better. But perhaps, given that people will act according to their own arbitrary values, libertarian values might be the values most conducive to social harmony in general since they would provide people with a set of default values which they can use when they interact with people they don't know. Libertarian values might be the most appropriate "default" set of values since they are probably the bare minimum necessary to ensure life on earth. Life on earth requires a) life, and b) access to scarce resources; following libertarian values would definitely assure these things. People can still subscribe to other values within their own value communities (such as females not being able to show their faces to men-strangers), but perhaps not with people outside their value communities. The problem with our current society is that certain values are legislated directly, or incentivized through social institutions, which have been imposed by coercive authorities; this creates an atmosphere where it is appropriate to impose one's own values on others, since it is efficient to do so. Political authority provides the means to do this; without established political institutions, I doubt this would be as easy. Without political institutions designed to impose certain values, I suspect that people's value sets would eventually shrink, and eventually approximate a libertarian value-set since it would be more socially efficient and less trouble.
I've just realized that over the course of writing this my views might have changed or become self-contradictory; please tell me what you think.