I do agree that having you wallet in your sig is akin to just casually asking every passer by if he can spare some change frankly. You don't expect anything but maybe he likes your outfit? Who knows! It's not really panhandling is it?
First off you're keeping an address in your signature, and not a wallet.
Second, it's not really comparable. If you ask every passer-by for spare change, you are actually bugging them about it. Leaving an address in your signature is more like sitting on a chair with a small sign next to you saying 'this is my address', and holding a speech (writing a post).