Also note that if you receive multiple transactions at an address, and then only spend some of the outputs, the remaining outputs will be left at an address for which the public key is known. Furthermore, if the wallet you are using does not use an unknown value for generating the signature then the remaining outputs become vulnerable.
Wait now I'm confused. Are you talking about output to a change address? I thought the change address would be safe because we are not sending from it.