Answering my own question above:

I am using Windows 10 on a machine with just 4GB RAM. Pagefile was set to 15GB which was enough for 2 GPUs, but not for 3. Setting it to 48GB fixed the issue.
As for the invalid shares, I don't know what originally caused them, as I have reinstalled Windows.