I have to admit I did not test it much on Nvidias - the grin code is more optimized for AMD since there are good CUDA miners for Nvidia around. What can happen - why it flips to 4G kernel - is that Windows takes some of the memory of the GPU even if there is no screen connected - and that is already enough to make the algo not work. We had that e.g. on 3G / 4G card when mining Beam as well - that runs well on Linux, but not on Win.
If you got the time please check nvidia-smi (comes with drivers) when no miner is running how much memory is reported as free. The current 8G grin code needs almost the full 8G (I am working on a new one, hope that saves more)
I've tried the latest lolminer on my 1080+2060 rig in Win10. The miner starts, initialises the cards (all in 4GB kernel) connects to the pool (grin31-eu.f2pool.com:13654) but disconnects 10 seconds later. It then starts a loop of connecting/disconnecting, and never actually starts mining.
EDIT: tried it in HiveOS too, no luck there either. Miner just doesn't start.