Try leaving out the rpcallowip directive anyways. What have you done to be sure that the miner is not able to connect via RPC?