But if LN's ledger system is as infallible as bitcoin's, then I guess there's nothing to worry about.
There is no ledger system, it's just two people with their own LN wallet. There could be a few hubs involved, but the only way to lose your money is if you lose your wallet.

lightning is still a ledger system.

we may not be publishing every transaction on the blockchain, but the pre-images we exchange are literally ledgers of accounts between channel participants. they're basically "private" ledgers that eventually settle on the blockchain's "public" ledger. (hence the privacy benefits of LN)