as quoted on http://networkingbunny.co.uk/documents/vmdirectpath :

Remember with ESXi you are always going to need a way of running the vSphere Client to manage it, now once running you may be able to live with running vSphere inside a guest for managing other guests. However for set up and when you need to alter your "desktop" guest you're going to need it on another machine.

Once (or before) ESXi is installed you need to go into the BIOS of your computer and ensure that the required extension is enabled (VT-d or IOMMU), wthout this ESXi will not be able to pass hardware through. If your platform supports it and is enabled you should now be able to go into the "Configuration -> Advanced Settings" section of vSphere Client.

Clicking "Edit" will provide you with a list of PCI/PCIe devices ESXi believes it can provide guests. You need to select the device which represents your graphics card. In my case it appears as "ATI Technologies Inc Optiplex 755" for one of my cards, then you'll need to attempt to match your USB controllers on the BUS to your physical USB ports. In my case I have two USB 2.0 controllers amongst six other USB 1.0 controllers, the difference of which is documented in the motherboard manual.

Unfortunately you will need to reboot the ESXi host in order to activate the passthrough configuration. I'm unaware if selecting all devices as possible passthrough options will cause you any issues, I specifically took the time to only make those required available.