Many good points n0nce.
Firstly, when you talk about energy efficiency of CPU vs ASIC, you are probably talking watts/hash. This isn't a valid metric because the algorithm determines how many hashes is "alot". A slow algorithm could provide just as much security at 100 hashes/second as a fast algorithm at 1 billion hashes/second. Hashes/second does not matter, what matters is how difficult it is to achieve 51% of the network hashrate.
Mining uses a lot of power. It has been discovered that the reason for this is (of course the value of the coin and the block subsidy) but also the low CAPEX/OPEX ratio of SHA256 hashing. In other words, the equipment to hash a tiny SHA256 input is very basic. So this frameword (an asic chip) can be produced cheaply and fed (with data and power) easily so that dozens or hundreds can live on a single chip. They can run reliably at high temperatures and speeds.
This leads to a massive ability for people with modest capital and lots of cheap energy to scale almost indefinitly.
On the other hand with CPU mining, CPU's are complex and difficult to feed (you often won't find more than one on a motherboard) and the capital (CAPEX) versus power cost (OPEX) ratio is very high. You simply can't easily scale up to 100 computers (the equivalent of 1 asic hashboard) very easily. Especially when these large miners can't dominate the market, everyone with a laptop from angola to zimbabwe will be competing with them.
For an example I am CPU mining as I am responding to you, browsing the internet, watching videos, there is no noticable slowdown. Yet this dual use is producing 1/3 the coins as an S9 miner would produce at a fraction of the power usage and heat production. So I don't think it would hurt the consumer market, in fact it would benefit it by not diverting extra fab capacity to unnecessary GPU chips or ASIC chips. Especially with the moore's law memory req scaling, consumers would always have, at the very least, last gen of CPU's on the secondary market to buy up.