Two transactions with different hashes and with overlapping inputs.
The definition is not entirely correct as a malled tx is not a double spend.
it will result in one though if the user has also tried to spend the change output of the original transaction that was voided by the malleable one. this seems to be happening quite a lot based on network chatter. probably because the client allows spending the change while it's unconfirmed.
I'm no expert on this stuff, but I'm wondering, could this behavior in the reference client be amplifying the effects of the malled TX attack as measured in the OP? i.e. When a client attempts to spend unconfirmed change that never gets confirmed because its tx got malled, is that also counted as a double-spend attempt?