Under Ubuntu, I can still use my GTX 1060 3Gb without any problem. Cards are using 2.7Gb out of 3Gb available. I suppose Linux takes 300Mb, is there a way to reduce this ?
You could disable the display manager and run your mining script over ssh, that should save a bit of memory.