I think you do: If you move coins from an address to another then I thought it was a taxable event ("work" is being done, like those crazy Jewish sabbath rules). Maybe if it's not in your control, but if that's the case then the argument should be move coins around at will, tax when converted into fiat.
They could not tell anyway. The way Bitcoin transactions work, the "change" gets sent to a different address than it was sent from in a modern wallet (indeed, it's not possible to distinguish the spending part from the change part without more transactional data and advanced analytics)