Yes, it applies to both parties in a channel. If either party tries to fraudulently close a channel by broadcasting a previous commitment transaction while the other peer is offline, then they have to wait whatever the two peers set the delay period to when they first opened the channel. This can be a maximum of 2016 blocks, which works out to 2 weeks at average block times. However, users are disincentivized to do this, as the other peer can come back online at any time and broadcast a penalty transaction, which means the dishonest peer would lose all their coins.