I thought that the main issue with space mining would be that it's way too expensive to go into space and perform the necessary work, which makes the operation a waste of money. However, there's actually a bigger problem: according to NASA, we're not there yet on a technological level. There is research and there is interest in this idea, but the tech just doesn't exist. Another thing that needs to be developed is a legal framework to establish the ownership of things mined in space.
Considering financial, technological, and legal obstacles, I think we should worry about the impact of space mining in the near future.