It's fine to use your Ledger for regular transactions due to security reasons mentioned above, but I (and many other people) agree that it's always good to have a hot wallet with funds you plan on using for convenience. I occasionally send funds from my SegWit Ledger to my Blockchain wallet for use online, and it works just fine for regular use. I've had this setup for three years now (though I only got the Ledger Nano a year ago, I previously used an airgapped computer for cold storage), and I haven't been hacked or compromised yet.

As a reminder, fees are extremely low right now. It wouldn't be a bad idea to combine low/dust inputs and consolidate them right now for lower fees in the long run, and perhaps start a SegWit address if you haven't already to reduce fees in the long run in the future if things change.