If I want to do more than 8 then I will need to rewire the SAS power. I believe that is what you did. Can you provide details in how to do that?
It's pretty easy. The SAS power connector has 10 pins. 5 commons / grounds (black). 3 12V (red) and 2 3V (yellow).
Assuming your cards only need 1 power connector, you are talking a total of 13 power connectors if 3 cars are installed in the system and 5 are on risers. If the GPU connectors have 2 each on 3 cables that's 6. You'll get another two out of the SAS connector. Leaves you short 5 power connectors unless you are going to use splitters.
I've been successful increasing density on my breakout boards by using molex powered risers that I can power three off one pcie power header. The pcie power header has 3 + and 3 negative at 12V. Each gets a 12v to 5v step down and then are passed to a molex plug. In essence I've turned one of the breakout board power connectors into 3 functioning molex connectors that will drive 3 risers saving 2 of pcie power connectors.
What version of Windows are you using?
Since it's quad CPU the DL580s are running Server 2016. However, the 4 GPU limitation I have experienced on both Server 2016 AND Win10. The DL360s and 380s run 10 since they are only dual proc.
It honestly sounds like a driver or firmware issue. Unfortunately with HPE servers you have to pay for driver updates. If you want to give it another try I would install windows and then install this service pack: