I think labor is wealth, whether little or much, wealth is also conceptualized as a type of property from low value to high value. For example, motorbikes, cars, gold, houses, land, farms... In fact, labor creates money and from there accumulates over time to have wealth. Whether you are rich or poor, you still have to work, the only difference is in the way you work.
I like the fact that you ended the statement by saying it all depends on the way you work hence I would have disagreed with you totally but perhaps you are also right in the end. Apparently labour isn't wealth but for one to be wealthy one has to labour. it's a two different thing altogether, when you labour in the wrong direction it can never create wealth for one to be wealthy you have make plans stick to those plans even when you are doing the hard labours hence you will be creating wealth but when you just work without direction it becomes a waste of effort and wealth can never be created out of such labour.