As far as I know, the difficulty of a block multiple of 2016 doesn't change if it takes more than 20 minutes to process. Such blocks are the only ones not affected by that rule in the testnet.
The difficulty of such blocks is calculated depending on the difficulty of the block immediately prior to them. If that block took more than 20 minutes, then this block could have the difficulty set to 1 as well, and so would the blocks mined after this one.
I said "could" because, if the previous 2,016 blocks took less than 2 weeks to process, then the difficulty could be slightly greater than 1.
It's actually because the block at the difficulty adjustment only looks at the difficulty of the block before it since we assume (for mainnet) that all blocks in an interval are all the same difficulty. However on testnet the difficulty can be one, so if the last block in the previous interval has a difficulty of 1, then the next interval will do that too. I have edited my post to reflect this.