No -- there is no transaction that is involved in this messaging.

Here's an example of how this messaging works, as I now understand it (please correct me if I'm wrong):

Let's say that you bought a widget from me, and you have already sent me the bitcoin to pay for it, but you haven't given me your mailing address so that I can send you your widget. Let's also say that we don't know each other and the only information I have about you is the bitcoin address from which you sent payment, and an email address that I think is suspicous -- it may or may not be you on the other end of the email address.

So, I ask you to send me a "signed message" that contains your mailing address.

You then go into Bitcoin-Qt, under "Sign message", and enter the bitcoin address that was used in the purchase of the widget, then enter your mailing address in the text of the message, then generate the signature. You then email me those three things -- the bitcoin address, the text of the message, and the signature.

When I receive those three things, I then enter them in "verify message" it it tells me if indeed this is a valid signed message. If it is, then I know that the mailing address in the message is from the person who owns the bitcoin address that paid for the widget.

It's kind of a brilliant feature for supporting anonymous yet trustworthy messaging.