It doesn't seem to be a site issue, as other publishers are getting clicks at that time. It's possible your source of traffic during that time is poor quality so not as many valid clicks.

I have noticed while promoting other sites and networks that many signups are low during certain hours, so it could be many variables.