...
I'm an Agnostic and a Libertarian who believes in living one's life for others' good, and the
accusations are clearly against my value judgement.
Unfortunately that phrase must be a lie. Libertarians are the most selfish pricks on the planet, and proud of it. In fact the enitre Libertarian ideology can be summed up in 4 words "Got mine, fuck you."
Self-interest is not selfishness.
A mature Libertarian knows that for his needs to be satisfied, he has to do good for others first. In a market, he improves the marginal utility for the counter-party so that the trade can happen mutually voluntarily.
They can only possibly say "got mine, f**k you" when they know they will die soon. Otherwise it's never good for his self-interest.