Average Joes are not going to have go track all there btc spending.  IRS has no way of tying you to a anonymous bitcoin address.  Nor do they want to waste time on it.  Pull your head out of your ass.