Another thing to keep in mind that unless the firmware fixes some glaring vulnerability or adds a feature you must have, then you can probably skip doing them.
Exactly. The newest Ledger Nano S firmware is 2.1.0 if I remember correctly. It introduces the needed support and necessities for Taproot. Other than that, it doesn't fix anything urgent or improve the user experience. Unless you want to use Taproot addresses with your Ledger HW, you don't need to perform the upgrade. It also decreases the already very limited internal storage of the device.