Privat keys are never to be extracted from hardware wallets.
I understand what you're saying about how the keys are stored and what Ledger did, but all HW wallets have secure elements, do they not? Those things I don't know much about except that the HW wallet manufacturer (as you stated) should never be able to access the keys stored in said secure element.
And a lot of pesky redditors were going on about no HW wallets being completely safe now, precisely because they all contain those secure elements, and they were talking about open-source ones like Trezor, Bitbox, and a bunch of others. I tend to listen to what's said here on bitcointalk, as you all know what the hell you're talking about--but the mood here is gloomy as well.
I feel paranoid.
Too little too late

Truely paranoid people (like me) wouldn't have trusted any third party device in the first place.
Yep. Unfortunately for me, I'm too damn trusting at times, and it gets me into trouble. Fortunately this didn't turn into a personal disaster, but it might yet for all of those people who still stand by their Ledger devices--and from the looks of it, there seem to be a lot of them.