No, he can't have access to your Bitcoin wallet just with your Bitcoin address since your Bitcoin address is a public address that you can send to anyone to receive Bitcoin through it, but anyone with your wallet private keys/wallet phrase will have full access and control over your wallet, you need to know the difference between both and know how to use and prevent yourself from every possible security risks.