It's pretty strange - I have 120000Mb virtual memory and 2 gtx 1070 ti have red error "out of memory". What is it?
Your memory should be enough for the miner to work properly. Maybe this is just a miner mistake that the developers will fix soon. I used to have a problem with lack of RAM on windows 10, I installed windows 7 and it worked.