Because your ip address goes directly to your internet connection, ie, your house, place of work, etc...  So even if your wallet address isn't traceable, you are still exposing your ip address to the network.  If someone is recording that information, they can follow a payment directly from you to the person you paid's address.  This information is not being recorded on the block chain, and it would take someone on the network logging this information to get it while it's happening, but still, we don't want any holes because holes get ripped open after a while.