If you earn money from manual labour, you're getting money from providing your skills and labour in order to do work; what you have to offer comes from you. If you instead earn money from pre-existing wealth, you're using your control of assets external to yourself that are required to do productive work in order to skim some money off the top. What makes you a better person to decide how those assets should be used than anyone else, such that you should be able to profit from controlling that decision?