When you meet enough people in your life, you get used to it. Some people are nosy and you learn to brush them off with witty replies. I don't get annoyed because I take pride in what I do, even if it earns next to nothing. I believe it's more to do with self-esteem than the question itself. Most of the time, the question is just an ice-breaker.
Maybe it seems like an ice-breaker to you because you are proud of what you do? I cannot say im proud of what i do really (like most), i've just tried to make money however i can for all my adult life. The question shows my potential and limitations but doesnt say much more about who i am. Most of my moral and ethical energy gets put into other areas of life. I do still make money legally and consdier what i do "morally" ok whatever that means.