If you earn money from manual labour, you're getting money from providing your skills and labour in order to do work; what you have to offer comes from you.
Really? So how much is the labor of a ditch digger worth if there are no shovels? How much is the labor of a ditch digger worth if there are no irrigation systems that require ditches? Some of the value comes from the laborer, but some of the value comes from the circumstances that make that particular labor valuable, which is external to the laborer.
If you instead earn money from pre-existing wealth, you're using your control of assets external to yourself that are required to do productive work in order to skim some money off the top.
Precisely what a ditch digger does. He controls a shovel that he did not make. He extends an irrigation system that brings water to farms he did not build.
What makes you a better person to decide how those assets should be used than anyone else, such that you should be able to profit from controlling that decision?
It has nothing to do with who is the "better person". You don't have to be a good person to deserve to profit from the invention of the shovel, the construction of farms, and whatever other environment you are lucky enough to be born into. A computer programmer doesn't have to earn the right to benefit from the technology of his society that he is born into by pure luck. Otherwise, every single person alive today would start out in greater debt than they could ever repay. The universe we are born into provides us its bounty for free, not fairly, but none of us really deserve what we get -- we just do because that's how the world works.