Not at all. You're not responsible for other people's feelings, despite what they might tell you.

Seems you're asking if it's wrong to be 'selfish'.