Thats right, its not the merchants. The weak link always has been and will continue to be exchanges.
If youre the customer of a brothel where you routinely cheat on your wife and you pay the same receiving address in btc every time and the only thing they sell is sex then youre in trouble. Its going to be easy for your wife to look at your joint wallet and see youve been paying a brothel. You have two choices, stop cheating on your wife or start paying in cash.
However, if you are buying online at a marketplace that sells drugs, guns, knitting supplies and ceramic collectible figurines then their could be some doubt about what you purchased because all of those things are paid for into the same address.
OTOH, If youre dealing with an exchange to buy btc or sell btc then you have to assume every government agent in the world was given all of that transaction info. Transacting at any exchange in the world eliminates, as you say, any degree of financial privacy.