This is mostly true .. in newer versions of Xen there is options for PCI passthrough, where it maps the physical device to a specific VM. So it is possible, but not very easy, and I wouldn't count on it not taking the whole box down if something went wrong.