At this point I'm pretty sure that there's a nasty bug with Polaris cards at least that shows itself on Linux Kernel 5.11 and later.
5.11 was when the fans suddenly got the zero rpm mode, so I'm sure there were some relevant changes made to the kernel.
Anyway, problem is that while the miner says that config clocks have been applied to all cards, even if I drop the mclock significantly in the config, all but GPU1 still hashes at the same rate and same power.
I noticed this because one of the cards started crashing which wasn't happening before. So I investigated and saw that power usage showed by the miner was significantly higher on kernel 5.11 and later.
I tried to drop the power usage by some means as an experiment and I just couldn't manage, no matter what.
So it's pretty clear to me that at very least, the miner thinks it applied the clocks from config successfully but it's not true for all cards.