Your freedom must be limited by the freedom of others. In order to enforce this freedom you need violence. You'll soon find that, as I tried to hint at earlier, you'll need a monopoly of violence. What you've suggested earlier is sometimes called the state of nature, but more accurately to be thought of as a state of war. Now... a monopoly of violence can be a rather depressing thing if it's randomly applied. So you'll want a rechtsstaat or some kind of rule of law. This is best achieved in a democracy. But not just any democracy, you'll want a representative democracy with separation of powers between the courts, the lawmakers, and the ruler.
I'm with you up to the democracy part, because I don't think freedom is best achieved in a democracy
if your version of democracy is that popular ideas are deemed moral because they're the most popular. That doesn't follow logically. If there are no absolute freedoms, democracy is rather worthless in my opinion. So it has to be a democracy that doesn't have the power to infringe on the freedom of the individual, and now we're back to a moral government.
Ok, so far, so good. Now you have a sound foundation for regulating society in a way that secures the freedom of every citizen. But what is freedom? Now we're back to what I mentioned in the beginning. Your freedom must be limited by the freedom of others. That is the extent and limit of your freedom... [snip] But to answer your question: if you retreat from your civic duties to be "left alone" you're infringing on the freedom of others and damaging this entire construct aimed at ensuring the freedom of all.
All this means is nobody's freedom takes precedence over any other person's freedom. No one can assign a "civic duty" to you without your consent, as such an action would be immoral. You are not born with an obligation to anyone, and any construct of such is artificial.