will a "hacker" need to have physical access to the device to hack/steal my private keys, or can he/she steal it remotely?
Ledger Recover gives Ledger or any hacker access to your keys over the internet.
Let's pretend that the device doesn't have the Seed/Wallet Recovery feature on.
Prove it.
That's the problem with Ledger's internet recover API. It gives internet access to the keys on a Ledger device. Sure, you may think you're safe because you didn't turn the feature on, but the code to give Ledger or hackers access to your keys is still on your wallet. Hackers won't care about the feature. They'll be hacking the code.
Once you've used a seed phrase on a Ledger device, there's no way to prove that seed is truly safe, so even if you decide to switch to a different device, you're just moving a seed Ledger or hackers may already have access to to another device.
A seed is no longer truly safe once it's touched a Ledger device. After using a Ledger device, the only way to be sure your coins are safe is to create a new seed on a different device and move your coins to that wallet.