I've been suffering from this since the early bitcoin-qt days when I started on bitcoin... huge gaps between download activity.. I have no idea why this happens. While the validation process has gotten faster, these huge gaps still happen. I would tell you to check on your connectivity (open proper ports) but then again as far as I know I have that set up properly myself and these huge gaps where nothing happens are still happening. And indeed when I close it and open it again it starts downloading pretty much right away.

Edit: unquoted OP because he doesn't like it