I believe this is the part that you're looking for: https://github.com/bitcoin/bitcoin/blob/master/src/pow.cpp#L54-L59

If the actual time is less than 1/4 of the expected time, then it will be equal to 1/4 of it, not any lower.
If the actual time is more than x4 of the expected time, then it will be equal to x4 of it, not any higher.