Someone that, over time, slanders others, will eventually make their way onto people's ignore list. If you really are as honest as others believe you to be (and as I believe you to be) (and as you present yourself to be) then no one will take such slander without proof.
Just because proof is an absolute defense to libel/slander does not mean that libel/slander lacking proof isn't a revocation of trust-worthy offense in any just society.
In a libel action, unless the plaintiff is a public official or public figure, the plaintiff does not need to prove financial damages. The common law has made a policy determination that the publication of defamatory statements in a transmittable, affixed form that is capable of widespread and simple dissemination will certainly cause damage and therefore damage to one's reputation or character is enough for a court to impose damages.
The libel was complete regardless if anyone will admit to believing in it.