No, that's not possible, your wallet address cannot serve as a means of stealing away your coins, if not, all of us would have also been a victim, make sure you you don't expose your seed phrase to anyone to see, then also make sure that you don't allow someone else to have access to your device you're using for wallet related apps, if they got hold of your device, they may see where you store your keys or make any transaction in your absence and they are irreversible once done.