I was actually wondering this myself the other day.  You can certainly hold physical PMs in your self-directed IRA.

I'd be very interested if someone here has looked into this further.