It happens continuously until I restart the miners.
EDIT - I updated to the newest cgminer and it starting happening again after roughly 45-60 minutes from when I started the miner.
This shouldnt happen anymore at all, I will run a queue test during next couple of hours and look at it when I wake up. Should be easy to figure out with the info at that time, eitherway let me know if it clears up at all.