Fully agree with you, i also owner of Ledger Nano S and it's nightmare to manage it sometimes. I also don't understand why the price of it is so high, when it just a simple token similar to RSA tokens.
Whilst I don't find hardware wallets practical or as secure as I'd like, it's certainly justified for the price of hardware wallets to be slightly higher than expected. Hardware wallets doesn't employ directly the technologies that are matured. The whole device is designed by them and its not just a "token". It does store the private key and sign the transactions on it.
RSA tokens literally just generates numbers and that's pretty cheap to make.