Given the price of large and more efficient panels needed to power a mining hardware that consumes a lot of energy, a small scale or medium scale miner might not have the amount of money to buy those. And it will take a longer time for that investment to return. Also, when the price dumps, it is another loss for the miner. Another is the halving of generated bitcoins for every block, there will be lesser and lesser bitcoin produced in the future. It's not that easy to start with buying a pricey solar panel.