Isn`t the point of the difficulty to be adjusted so that it tries to stay around 1 minute? In such case it will take some time before it adjusts to 1 minute as clients come and go.