Productivity and/or hard work is generally inversely related to wealth.
Hmmmm. So then does it hold that the harder I work, and more productive I am, the less wealth I will accumulate? Or does the inverse of that hold, so that the less I work, and the less productive I am, the greater my fortune will become?
Certainly you can't believe those things?
Read the quote by Smith again until you understand what he's saying. This isn't theoretical, this is how things actually arrange themselves without active intervention.
In a free market or anything close to a free market, the hardest working people are those on the lowest rung of the ladder. The guy that works the hardest is the guy in the sweatshop that gets paid pennies per week. The entire system depends on this guy, because it is his cheap labor and hard work that allows everyone else to reap the massive profits of his labor. This man does more work than anyone, but paid less than 1/400 of even a minimum wage US worker.
On the flip side of the working world we have the corporate executives that are paid, on average, 300-500x more than the US workers (not international, sweatshop workers, which they are paid many hundreds of thousands of times more than). Do they do 300-500x more work than the guy standing behind the cash register? Absolutely not.
Then we move into the non-working world - the owners of the means of production, land, buildings, etc. These people do literally next to nothing, but reap the most rewards of anyone... for doing absolutely nothing. They don't need to lift a finger to do labor, but they still collect ungodly amounts of money simply because of what they own, many hundreds of millions of times more money than the guy busting his ass for 15+ hours per day, 7 days per week in the sweatshop. These people are not at all vital to the system, only their assets are.
The bottom line here is that financial reward in a free market system is NOT tied to how hard one works. That myth is just that, a myth. Financial reward is based on power, and power is based on the control of the means of production. Want it get rich? Hard work is not how you do it. You need to strive for control of the means of production and/or the power to control people.