Whenever a hardware wallet wants to support coins other than bitcoin, then it has to introduce new code to do so. Any new piece of code poses a risk, as it is impossible for the developers to discover every possible way in which that code can be used to interacted with. For example, there was a (now fixed) vulnerability a few years ago with Ledger devices, in which an attacker could trick you in to signing a bitcoin transaction, even though the hardware wallet showed that you were signing a transaction on an altcoin. If the Ledger device did not support altcoins at all, then such an attack would not be possible.

The less code of any type on a device, the more secure it will be. Whether or not having bitcoin only firmware actually prevents any future attacks, though, is obviously unknown.

The choice is up to you. There are plenty of people out there who are only interested in holding bitcoin, but there are also plenty of people who want to buy dozens of altcoins. I would, however, point out that >99% of altcoins are either useless vaporware or outright scams.