Dunno if this project is dead or what... but if you continue, for the software side, you might want to consider a Xen based platform.  Xen is without doubt an ugly stepchild in the virtualization world, but it's been used extensively for PCI passthrough -- indeed, if I'm not mistaken, it even supported come crude virtual PCI passthrough capability before hardware IOMMUs came to market.

However I will warn you that Xen adds some complexity and if you're already struggling you might not like it.  Another approach would be to get rid of Proxmox and use regular linux.  If I were doing this myself, I would probably build it as a Gentoo system with a hand-configured kernel -- but again that's kinda complex.

I've tried proxmox and found it to be pretty buggy and fragile.  Great concept, but they just need to do more work on their implementation.  Everything seemed to be half-way implemented, so as soon as you deviated from the simple use-cases they designed it for, everything fell to pieces.  Actually everything fell to pieces even when I tried to follow the simple guides in their wiki.

Proxmox is basically debian with an openvz kernel and a bunch of Red-Hat cluster software.  You don't need the extra complexity that openvz or Red Hat cluster suite bring into the system -- you might have better luck just running ubuntu and kvm.  It's not that hard -- just follow the guides in the wiki and you'll be 90% of the way there.

I understand the allure of a GUI like Proxmox, but often the "simplicity" offered by such tools is illusory and ends up acting more like a "complexity loan" -- installation will be easy, sure, but then try to actually /do/ anything and you will spend more time than you saved during installation working around all the bugs and limitations of whatever mysterious pile of software is sitting underneath that pretty GUI.