So...when one can build their own blocks, they can also set the timestamp in those blocks.  When one can set the timestamp, we can have two blocks with a real-world 3 hours of time between them, but a 38 second reported block time.  And when the wallet's block time testing isn't working as it should, blocks that are earlier than the most recent in the chain can be accepted as valid.  And...here's the important point to all of this:  The diff algo uses the encoded block time to adjust the difficulty - it uses the reported block times, not the real-world times.  That's why we can have widely spaced blocks - that should result in our difficulty rapidly decreasing - yet find our difficulty rising.  And while our algo is designed to rise rapidly when hash power increases, it really jumps when someone tosses in blocks with timestamps that go 'back in time.'