Quote from: z-axis on October 25, 2017, 04:11:02 PM

Quote from: woodl1 on October 25, 2017, 12:16:41 PM

Max GPUs i have in one rig is 13. In case this doesn't help we need to see nvidia-xconfig output and xorg.conf that it makes, so we can try to find what's wrong.

Also you can try to reflash the latest version of nvOC to usb stick, edit 1bash at windows partition (don't forget to enable headless mode) and make the first boot from usb stick without monitor attached. After the first boot in headless mode you need to wait for autoconfiguration and auto reboot. Then use network scanner or see DHCP leases on the server side to find the IP of your rig (in most cases it will be the same as in previous attempts), ssh to it and see if xorg was correctly configured.

Thank you very much for your help!
My rig is now running with all those settings: New USB, Headless, no monitor attached, know IP, have SSH and mining. Here's the weird thing: See below for my current xorg.conf that is running and I'm getting all 19 GPUs mining but only the 1st one is OC'd. What's weird about it is that:

When I run

Code:

sudo nvidia-xconfig -a --cool-bits=28 --allow-empty-initial-configuration

I get a new xorg.conf that does two things:
Bus IDs lose the "@" symbol, and mining does not work.

Any ideas why some BusIDs use the @ while most I've seen don't, and why this seems to be allowing mining to run?
Also weird is that if you notice, there are only BusIDs for 0-3 (4 and 5 are missing), then it continues from 6-11 and that's it.

What are your thoughts on this?

Current xorg.conf:

Code:

Section "ServerLayout"
Identifier "layout"
Screen 0 "nvidia"
Inactive "intel"
EndSection

Section "Device"
Identifier "intel"
Driver "modesetting"
BusID "PCI:0@0:2:0"
Option "AccelMethod" "None"
EndSection

Section "Screen"
Identifier "intel"
Device "intel"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:1@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:2@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:3@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:6@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:7@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:8@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:9@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:10@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

Section "Device"
Identifier "nvidia"
Driver "nvidia"
BusID "PCI:11@0:0:0"
Option "ConstrainCursor" "off"
EndSection

Section "Screen"
Identifier "nvidia"
Device "nvidia"
Option "AllowEmptyInitialConfiguration" "on"
Option "IgnoreDisplayDevices" "CRT"
EndSection

current lspci:

Code:

00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM
00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (
00:02.0 VGA compatible controller: Intel Corporation Device 1902
00:14.0 USB controller: Intel Corporation Device a2af
00:16.0 Communication controller: Intel Corporation Device a2ba
00:1b.0 PCI bridge: Intel Corporation Device a2eb (rev f0)
00:1b.5 PCI bridge: Intel Corporation Device a2ec (rev f0)
00:1b.6 PCI bridge: Intel Corporation Device a2ed (rev f0)
00:1b.7 PCI bridge: Intel Corporation Device a2ee (rev f0)
00:1c.0 PCI bridge: Intel Corporation Device a294 (rev f0)
00:1c.5 PCI bridge: Intel Corporation Device a295 (rev f0)
00:1c.6 PCI bridge: Intel Corporation Device a296 (rev f0)
00:1c.7 PCI bridge: Intel Corporation Device a297 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Device a298 (rev f0)
00:1d.1 PCI bridge: Intel Corporation Device a299 (rev f0)
00:1d.2 PCI bridge: Intel Corporation Device a29a (rev f0)
00:1d.3 PCI bridge: Intel Corporation Device a29b (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device a2c8
00:1f.2 Memory controller: Intel Corporation Device a2a1
00:1f.4 SMBus: Intel Corporation Device a2a3
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connecti
01:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
02:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
03:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
04:00.0 PCI bridge: ASMedia Technology Inc. Device 1187
05:01.0 PCI bridge: ASMedia Technology Inc. Device 1187
05:02.0 PCI bridge: ASMedia Technology Inc. Device 1187
05:03.0 PCI bridge: ASMedia Technology Inc. Device 1187
05:04.0 PCI bridge: ASMedia Technology Inc. Device 1187
05:05.0 PCI bridge: ASMedia Technology Inc. Device 1187
05:06.0 PCI bridge: ASMedia Technology Inc. Device 1187
05:07.0 PCI bridge: ASMedia Technology Inc. Device 1187
06:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
07:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
08:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
09:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0a:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0b:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0c:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0d:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0e:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
0f:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
10:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
11:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
12:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
13:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
14:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)
15:00.0 3D controller: NVIDIA Corporation Device 1c07 (rev a1)

as for BUS IDs naming, I don't have ideas why your rig using @, but if it works - let it be so

The thing with 4-5 pci bus skipped is because other devices are using these buses, it's normal (you can see it with lspci - "04:00.0 PCI bridge: ASMedia Technology Inc. Device 1187" and so on).

What I want to point in your current xorg.conf is that there is no Coolbits option in each Screen section. This option has to be set to overclock GPU, I suppose, so try out what fullzero suggested:

Code:

sudo nvidia-xconfig -a --cool-bits=12 --allow-empty-initial-configuration
sudo reboot

and if it doesn't help, try to add this option to each Screen section in your xorg.conf except the screen section with intel device.

Code:

Option "Coolbits" "12"

Here's xorg.conf from one of my working rigs, it got 7 GPUs, all P106-100.

Code:

# nvidia-xconfig: X configuration file generated by nvidia-xconfig
# nvidia-xconfig: version 384.59 (buildmeister@swio-display-x64-rhel04-01) Thu Jul 20 01:03:28 PDT 2017

Section "ServerLayout"
   Identifier "Layout0"
   Screen 0 "Screen0"
   Screen 1 "Screen1" RightOf "Screen0"
   Screen 2 "Screen2" RightOf "Screen1"
   Screen 3 "Screen3" RightOf "Screen2"
   Screen 4 "Screen4" RightOf "Screen3"
   Screen 5 "Screen5" RightOf "Screen4"
   Screen 6 "Screen6" RightOf "Screen5"
   InputDevice "Keyboard0" "CoreKeyboard"
   InputDevice "Mouse0" "CorePointer"
EndSection

Section "Files"
EndSection

Section "InputDevice"
   # generated from default
   Identifier "Mouse0"
   Driver "mouse"
   Option "Protocol" "auto"
   Option "Device" "/dev/psaux"
   Option "Emulate3Buttons" "no"
   Option "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"
   # generated from default
   Identifier "Keyboard0"
   Driver "kbd"
EndSection

Section "Monitor"
   Identifier "Monitor0"
   VendorName "Unknown"
   ModelName "Unknown"
   HorizSync 28.0 - 33.0
   VertRefresh 43.0 - 72.0
   Option "DPMS"
EndSection

Section "Monitor"
   Identifier "Monitor1"
   VendorName "Unknown"
   ModelName "Unknown"
   HorizSync 28.0 - 33.0
   VertRefresh 43.0 - 72.0
   Option "DPMS"
EndSection

Section "Monitor"
   Identifier "Monitor2"
   VendorName "Unknown"
   ModelName "Unknown"
   HorizSync 28.0 - 33.0
   VertRefresh 43.0 - 72.0
   Option "DPMS"
EndSection

Section "Monitor"
   Identifier "Monitor3"
   VendorName "Unknown"
   ModelName "Unknown"
   HorizSync 28.0 - 33.0
   VertRefresh 43.0 - 72.0
   Option "DPMS"
EndSection

Section "Monitor"
   Identifier "Monitor4"
   VendorName "Unknown"
   ModelName "Unknown"
   HorizSync 28.0 - 33.0
   VertRefresh 43.0 - 72.0
   Option "DPMS"
EndSection

Section "Monitor"
   Identifier "Monitor5"
   VendorName "Unknown"
   ModelName "Unknown"
   HorizSync 28.0 - 33.0
   VertRefresh 43.0 - 72.0
   Option "DPMS"
EndSection

Section "Monitor"
   Identifier "Monitor6"
   VendorName "Unknown"
   ModelName "Unknown"
   HorizSync 28.0 - 33.0
   VertRefresh 43.0 - 72.0
   Option "DPMS"
EndSection

Section "Device"
   Identifier "Device0"
   Driver "nvidia"
   VendorName "NVIDIA Corporation"
   BoardName "P106-100"
   BusID "PCI:1:0:0"
EndSection

Section "Device"
   Identifier "Device1"
   Driver "nvidia"
   VendorName "NVIDIA Corporation"
   BoardName "P106-100"
   BusID "PCI:2:0:0"
EndSection

Section "Device"
   Identifier "Device2"
   Driver "nvidia"
   VendorName "NVIDIA Corporation"
   BoardName "P106-100"
   BusID "PCI:4:0:0"
EndSection

Section "Device"
   Identifier "Device3"
   Driver "nvidia"
   VendorName "NVIDIA Corporation"
   BoardName "P106-100"
   BusID "PCI:5:0:0"
EndSection

Section "Device"
   Identifier "Device4"
   Driver "nvidia"
   VendorName "NVIDIA Corporation"
   BoardName "P106-100"
   BusID "PCI:6:0:0"
EndSection

Section "Device"
   Identifier "Device5"
   Driver "nvidia"
   VendorName "NVIDIA Corporation"
   BoardName "P106-100"
   BusID "PCI:7:0:0"
EndSection

Section "Device"
   Identifier "Device6"
   Driver "nvidia"
   VendorName "NVIDIA Corporation"
   BoardName "P106-100"
   BusID "PCI:9:0:0"
EndSection

Section "Screen"
   Identifier "Screen0"
   Device "Device0"
   Monitor "Monitor0"
   DefaultDepth 24
   Option "AllowEmptyInitialConfiguration" "True"
   Option "Coolbits" "28"
   SubSection "Display"
   Depth 24
   EndSubSection
EndSection

Section "Screen"
   Identifier "Screen1"
   Device "Device1"
   Monitor "Monitor1"
   DefaultDepth 24
   Option "AllowEmptyInitialConfiguration" "True"
   Option "Coolbits" "28"
   SubSection "Display"
   Depth 24
   EndSubSection
EndSection

Section "Screen"
   Identifier "Screen2"
   Device "Device2"
   Monitor "Monitor2"
   DefaultDepth 24
   Option "AllowEmptyInitialConfiguration" "True"
   Option "Coolbits" "28"
   SubSection "Display"
   Depth 24
   EndSubSection
EndSection

Section "Screen"
   Identifier "Screen3"
   Device "Device3"
   Monitor "Monitor3"
   DefaultDepth 24
   Option "AllowEmptyInitialConfiguration" "True"
   Option "Coolbits" "28"
   SubSection "Display"
   Depth 24
   EndSubSection
EndSection

Section "Screen"
   Identifier "Screen4"
   Device "Device4"
   Monitor "Monitor4"
   DefaultDepth 24
   Option "AllowEmptyInitialConfiguration" "True"
   Option "Coolbits" "28"
   SubSection "Display"
   Depth 24
   EndSubSection
EndSection

Section "Screen"
   Identifier "Screen5"
   Device "Device5"
   Monitor "Monitor5"
   DefaultDepth 24
   Option "AllowEmptyInitialConfiguration" "True"
   Option "Coolbits" "28"
   SubSection "Display"
   Depth 24
   EndSubSection
EndSection

Section "Screen"
   Identifier "Screen6"
   Device "Device6"
   Monitor "Monitor6"
   DefaultDepth 24
   Option "AllowEmptyInitialConfiguration" "True"
   Option "Coolbits" "28"
   SubSection "Display"
   Depth 24
   EndSubSection
EndSection