Bitcoin mining is a costly business. If you do not have enough capital, it will not be profitable. Nowadays miners are carrying out their expenses by minimizing their costs.
But difficulty of the mining business does not adversely affect the use of bitcoin. If somebody leave mining, the level of difficulty decreases. Then mining would be less costly.