*snip*
I installed new cuda and affined the threads to one cpu-core, working now. Thx for the tip and your work

No prob, glad to help.
So I've tried this on my "regular" GTX 1080's and I'm getting hashrates just below 700 per GPU if I use the automatically generated parameters and the ETHlargement pill on the new Cryptonight/r algo.
Has anybody got better results? Any recommendations for tweaked parameters? I've checked that XMR-Stak bench site but they've got very little results for 1080's...
Thats about what I get on my reference style 1080.
you already checked my main recommendation:
https://www.xmrstak.com/You can look up other users settings for optimizations; I did that for mine and only had to trim a tiny bit on my 1070; I just left my 1080 on auto since it rarely mines now and is in my daily use PC.
Look at any 1080 ones you see; as well as 1070; they will probably be similar; just take into count when transposing the 1080 having ~8-10% more computing power.