Let's say we only have one ledger block, or whatever, or UTXO commitment thing. If we don't know the blocks before that, other than just headers, we can't verify that people are building on valid blocks. Someone could have made a fraudulent ledger block, and built on top of it. If people don't catch this, and never check contents before that, they have successfully attacked the network.

I think of it as SPV++ security: We are saying the ledger blocks/UTXO commitments are "secure" based on how deep in the chain they are, rather than looking at height. The only way to know the chain is valid is to start 100% from the beginning, and work your way through until you reach the current height.