That is what I always thought myself, but now the big concern we have to contend with is that the IRS is jumping in front of Bitcoiners and saying they want a piece of the action.
This is by no means unique to bitcoin. The IRS want to stick their noses in
everything. Bitcoin may seem different since it is actually a currency in its own right, but since the IRS class it as property, it will be taxed as property.
Let's say we are trading basketball cards. I swap you a rare Kobe Bryant card for a rare LeBron James card. No fiat changes hand. No values are even discussed. As far as the IRS is concerned, we both sold our cards to the other person for fiat, and then used that fiat to immediately buy the other person's card. We both owe taxes on the capital gain made, which is the difference between what we bought the card for originally, and what its market value was when we traded it to each other.
Do people report such trades and pay taxes? Largely no, but there is no publically accessible ledger of all basketball card trades.