No one else can withdraw money from your wallet without your help. He doesn't have the ability to withdraw money from your wallet unless you help him. You are completely worry free as no one can withdraw money from your address even though your address is visible to everyone. I tell you for example we work on signature campaigns we get paid weekly if someone could transfer my dollars everyone would take out. But keep your wallet seed phrase very safe so that no one can steal it. You are completely worry free if no one can steal the wallet's seed phrase.