It's not rocket science. If your fan has to push through a bunch of restrictions, it's going to make more noise than without restrictions. So removing the exhaust side grills should help a very little bit on the noise side. And it will allow for better air flow, or the same air flow at slower fan speed. Furthermore, a single fan pushing air through a tightly packed case is going to have to turn faster to provide sufficient airflow. Adding a fan pulling air on the exhaust side should allow me to turn them down to around 4000 RPM instead of the stock 6000 RPM.

What I am planning is similar to this:
https://www.silent-miner.eu/

My big 6" Vortex squirrel fan will be pushing air in while the stock fans will be relocated on the exhaust side to pull air out with a fan controller on them. I'll have to test fan speeds to see what works just as well as the stock configuration.

And of course none of this is going to completely silence the rig.  It will still be noisy, albeit less noisy. But next I will do similar to what philipma1957 did. I will build a box with a 1 foot layer of carpet pieces around the miner - a soundproof box of sort. Just try to run a vacuum cleaner rolled up in a pile of carpets and see how quiet it gets.

But in truth, even if I just replace the stock fans with my squirrel fan, that would already push more air and be a great deal more quiet. I had my squirrel fan running all night at full speed right next to my bed last night. Slept like a baby too but I will look to make it even more quiet.