I thought the same thing but the problem is that nvOC is an easy to use mining system. I think it won't be easy to use for lots of people if you remove gnome desktop or another desktop interface. A lighter interface desktop would be a better solution.
I always prefer to build my mining linux myself. But for nvidia, I wasn't able to control undervoltage/overclocking. That's why I still use old nvOC. But from what I remember, these functions required from me to install X11 along the computing drivers. So I am not sure if desktop makes it so fat and not a required dependencies. Without them the system works, but was no fine tuning possible. I may be wrong.