What's coming down from his panels aren't 12V, they're likely strings somewhere in the 150-300V range. Even individually, larger panels like this individually are above 25V themselves. Not to mention the solar power isn't constant enough; you'd be varying input voltage constantly. You need the inverter to smooth it out and make it a clean and stable power.
Hack some cheap server PSUs to perform MPPT, then parallel the output with unmodified server PSUs to get a constant voltage. Hack the load share to give preference to the solar PSUs, with the normal PSUs supplying the remainder. It will take a lot of electronics knowledge to do it.
Unless you live in the desert, even with MPPT, the voltage won't be stable enough. The moment a cloud hits a panel, you're going to see voltage swings without a buffer. If the 5-8% loss is of that great concern, having a battery as a voltage buffer is the safest way of doing it. Not to mention the PSU and GPU don't keep a stable draw, it varies constantly. The MPPT needs something to power track against. If both the power source and load are constantly varying, that's not the best.
I think the OP is on the best route. Operating large solar systems at 12V is NOT cost effective. The cable sizing required to bring the amount of amps from any length of distance at 12V is outright impossible and certainly not cost conscious. Keep in mind there are MANY factors in the overall efficiency of a solar system, cabling and inverter are certainly big ones. Having the solar system bring in the high voltage (the whole point of strings) and invert it (that inverter is likely 98% efficient) to clean 120VAC is worth the few % lost.
I was a solar designer and installer for almost a decade and the question of can I do this right off the panel came up very often. I have a very similar mining/solar set up myself.