What agreement did you make on returning the money when they ask for it? I'm asking because of this scenario. Short summary: in that case the escrowed funds were asked back shortly after someone deposited an amount large enough to exit scam.
Escrow only makes sense if there's at least 1 week "cool down period", during which you publicly announced that you'll pay back funds in a week.