It is not possible mate since you are the owner of the wallet and let's just say that your wallet is non-custodial wallet as it is the most recommended wallet compared to custodial wallets. I think you might heard or read somewhere about this phrase "NOT YOUR KEYS, NOT YOUR COIN" which is true. Anyway, that person won't be able to access the wallet unless you told him about your seed phrase or mnemonic phrase which is used to access the account. There are two things that you should know, public address is what you will use to receive payment or when you are paying someone and the seed phrase is what you shouldn't share to any or private in short. Take note, only those who has access to the wallet are the only one who can withdraw BTC from the wallet unless someone have access of your wallet either you shared it (which is not recommended or the wallet is compromised through hacking/phishing).