... so they just removed everything not needed for mining, no video output etc, high power VRMs for insane clock speeds, loads of high-speed memory (because they are such a big company and buy tons of it they get it cheaper than anyone else), so they run 10-100x more efficient (both MH/s/J and USD/(Mh/s)) than the GPUs they sell.
Unlikely. 1.05x more efficient, I'd believe. I don't know how you come up with 10x-100x...
GPUs are already built mainly for GPU-compute. People use them in supercomputers today. Yes, in theory someone like AMD could spin a GPU with some dedicated generic crypto accelerators, but you always have 2 ways to go. Flexibility to accommodate different workloads, or flat out performance on a pre-determined workload.