What are your overclock settings for both core and memory clock? One possible reason why your rig keeps restarting is because the GPUs are failing which could be due to bad overclock settings. Try dialing them down a bit.
could be 4gigs of ram in windows.
4 GB ram should work fine for a simple mining rig. Im running 13 GPU rigs with 4GB RAM and with mixed rx VEGAs and 1070s and they work fine.