I can't get more than 4 working, even with the GPUmod to allow more cards. I'm thinking it's just a crappy chipset...
Sounds like you do not have 'Above 4G Encoding' enabled.
When enable 4G Encoding should use ONBOARD Video or PCIE Video?
4G Encoding should work with 3 GPU? (FOR TEST then try with 5)