You're correct. It will be very difficult for the IRS to ever link that to you. However, should you want to buy something with your BTC or convert your BTC into fiat, and the IRS links your purchase / trade back to you, then discovers you've avoided paying taxes on your BTC trades, you're looking at a tax evasion charge.
Everyone keeps citing that we'd be evading taxes, while completely missing the fact that cryptocurrency makes enforcement of the tax laws 9999999 times more complex and difficult.
So the IRS is tracking Bitcoin transactions? Okay cool, there are like 60 other cryptocurrencies we can use in that case, and more are being created every single day.
Have fun expanding the size, scope, and budget of the IRS by a factor of 60.
The harsh reality is this:
Very soon the IRS will be more expensive to run than the amount of money they bring in for the government. They are being rendered obsolete, and will go the way of Blockbuster Video. Ditto the NSA.