I want to ask you that did you gave him your wallet private key or only the public address? It's a very easy thing that How can he withdraw btc from your wallet if he doesn't have access to your wallet. Already, many people have given the answer, but I am still saying that once the confirmation of the Bitcoin transaction is done, you can never withdraw money from that wallet until you know the private key.