Also note that if you receive multiple transactions at an address, and then only spend some of the outputs, the remaining outputs will be left at an address for which the public key is known. Furthermore, if the wallet you are using does not use an unknown value for generating the signature then the remaining outputs become vulnerable.
Wait now I'm confused. Are you talking about output to a change address? I thought the change address would be safe because we are not sending from it.
Outputs that are received at a Bitcoin address are individually spent in their entirety. If you are using a wallet with coin control you have the ability to choose which outputs are spent in a transaction, and can therefore make sure that all outputs received at a particular address are spent together in a single transaction. If you are not using coin control, then it is possible that some outputs will be spent separately from others that were received at the same address.
Bitcoin doesn't distinguish between change addresses and receiving addresses. At the protocol level, it's all the same.