The problem is as a wallet address is associated to a person everything you've done and are yet to do can be traced. There are ways to regain anonymity but it wont be easy for the average user. I don't see this as a real problem today but in the future when this information is easily accessible companies will use your personal private information against you. There have been situations like this already; Target for example found out a mans young daughter was pregnant due to her buying patterns matching those of a pregnant woman before he even knew.

Unless a solution to these situations is found this could become a real problem. I can see it effecting insurance to mortgage rates and a multitude of other things.