Get receipts for your face to face transactions. Problem solved.
If you spend some coin face to face and don't have a receipt, and instead you set the value from what the main bitcoin exchanges were using as the rate, would the IRS accept that? If you got, and reported, a higher value, wouldn't the IRS be even less likely to dispute it?