What do you mean by your bitcoin server? your bitcoin qt client acting as a node? i think you need to open some ports for that to work.