So, I've been on the hunt for a good MB to do 6+ GPUs with. Awhile back on a hunch I bought the ASUS Z270A PRIME -- 7 PCIe slots and two m.2 slots - I thought maybe I could get 9 GPUs on it. But I was restricted to 4 with the bios that shipped with it.
At this point, however, I have gotten 7 GPUs on it-- the max I have space for right now.
Sharing the details here in case it is useful for someone else:
Here's what I found:
-- Had to flash the BIOS. Board shipped with 5.0.0 and I updated it to the latest (7.0.0 I think)-- this is necessary to get the "Above 4g decoding" option. Which appears on the boot menu in the bios.
-- PCIe switched to Gen2
-- Didn't make other changes.
WARNING WARNING WARNING
When I flashed the BIOS the onboard network hardware stopped working with linux. this caused a bunch of problems.
I had to run this utility to refresh the network drivers ROM to make things work:
https://downloadcenter.intel.com/download/19186Current setup for this board-
-- Ubuntu 16.04 server
-- 7x NVIDIA 1070 8GB Cards
Working fine. (And about %10 more hash under linux than under Windows.)