Also, when people look at the blockchain, all they'll see is someone sent this amount to this address. They'd not know what that is for unless there's a message included.
They know if they are a party to one of the transactions. They can look at linkages between the transaction they know about and others. For example, if you run a business, your landlord or other vendors can look at the source of the funds you use to pay them. Likewise your customers can see where the coins they are sending you are going. They can also often (if not always) find other customer payments to you, since those payments will often be aggregated to make payments to vendors, salaries, etc., again establishing linkages.
Much has been written about blockchain analysis. It is not as trivial as you suggest.
And you could easily change address anytime you want.
Changing addresses does not help you because transactions are linked, publicly, on the block chain. If you want to break those links you need some kind of mixing, coin exchange, zero-knowledge proofs, etc. which brings us right back to the realm of "anonymous" technologies.