I would think it could only go up.  Won't it just take longer to reach the next difficulty level if mining activity drops?