Money is a way to transfer value.
You give a service or goods to someone, and you receive money which you expect to be able to give back later in exchange for some other good or service. It's nothing but a convention between people. Therefore, all money you own is some kind of a debt society has towards you.
If you lose money, you relieve society from this debt. It's just as if what you did to obtain this money, you did it for free.
I don't see how people working for free could hurt society.