Think I figured out the issue along with some help

Seems memory could be the problem. This rig only has 8gb of Ram. When I increase page filing to 65,000 I saw good improvements. Instead of the other 5 GPU's all being below 5Mh/s now it's only 1.
*Edit*
Scratch that. Thought the page file might have fixed it however getting same results as I let it run :/
Im not suggesting or trying to force a miner switch on you, but is it possible to run TRM ethash on the first 5 gpus instead of Claymore and test again? It just makes it easier for me to reason about the underlying issue(s) since I know exactly what happens in the codebase for TRM but not for CM. Or am I stupid and we cant do Callisto?