One downside is that you won't see any evidence if things aren't working. In fact I'm not sure how the miners work exactly but it seems like they might not failover under some conditions if they don't have the opportunity to try get new work and have it fail, so your miners might continue mining on a down pool.