As true as 'ledger does not have your keys' may be at this point. Remember that much of their hardware and software is proprietary and closed source. We don't fully know what the software and hardware does or is capable of doing.
I totally agree with this one, I'd rather keep it simple save your keys on a small piece of paper. Put in a secured and safe storage. Its not that I can't trust those third party protection but its better to be safe than sorry right?