Can Bitmain make ASIC similar to the AMD GPU, but with more instruction set for mining?
What you're describing is an FPGA. Even they can be forked out with the proper algo.
With the proper algo, ASIC's can be rendered on an even keel for the most part with GPU's. Like 1.1 to 1.0 OR 1.2 to 1.0 ratio. Mainly due to "mapping."
Then it would mean BITMAIN would more or less be replicating GPU's and would have to try to compete in manufacturing costs with AMD, NVIDIA, INTEL, etc... at that point.
Have a look at PROG Proof of Work. ESPANSE has already said they will use this algo:
https://www.youtube.com/watch?v=P6B-ZCyP6K8&t=1s