I thought of a way to do "secure SSL" messaging on the blockchain to avoid direct connections between payment and public addresses. I'm not sure if that would solve your problem or not, but canceling or refusing a tx would require sending your cancelation to all miners, and that they accept your cancelation rather than mining the original tx into a block. It's probably not practical to do it that way for several reasons, and it would definitely require a protocol change.
"Canceling the transaction" sounds like "reversing" a transaction.
This is not the intent. This would be the destruction of bitcoin, IMO.
The intent is that the Application Layer will check incoming TX, and either
A) just leave it unredeemed for the Customer's future recovery attempts via Sending entity
OR
B) redeem it, and return it with a message in the OP_DROP field.
I do not see why a protocol change would be necessary or desirable