Also note, there is no universal exact time that all miners and pools are using.  Because of this, there may be some miners and/or pools that use the timestamp as an additional nonce. The timestamp in the block header can vary quite a bit. So, that 5 seconds might not actually be 5 seconds.

So, some of the "delay" that you are seeing may be attributed to:

1. Time for the first completed block to be propagated from the miner that "solved" it, to the pool that miner was participating in, to the pool that will solve the second block, to the miner that will solve that second block.
2. Differences in clock time between the two pools that build the two blocks.
3. Time for the second pool to verify the first block, and then build the second block.
4. Any alterations made to the timestamp in the process of solving the second block.