Yes, but I guess what @oryhp says is that if you communicate without a secure connection you can't be sure there isn't someone spying on you without you knowing it. Sure, he can take the money, but what's more valuable? Depends on your threat model. 
I was assuming a secure connection. Nothing works if you don't exchange the receiver address securely... silent payments are slightly better in the sense that they require a single secure exchange as opposed to having a secure exchange for every receiver address. Securing connections shouldn't be that hard today.