With the default assumeValid, it's not fully verifying from scratch, since it only verifies historical scripts and signatures of the last few years. You're trusting the developers' claim that all earlier scripts are valid too.
Since I read about that, I always felt that it goes agains "verify, don't trust". Since you're downloading all blocks anyway, how hard can it be (for your computer) to verify everything it downloads?