FWIW, bitcoinjs has several times had bugs where it would generate incorrect pubkeys which no one could sign for. Personally I would be extremely wary about using it for any serious application.

Most other implementations of bitcoin's crypto are extremely scary-- e.g. contain very little to no testing, had no rigorous review, were subject to no formal validation, are not implemented in a fault tolerant way, and are (IMO) unfit for high value usage compared to Bitcoin Core.

I'm aware of at least one non-public incident where over 10,000 BTC was lost due to less thoughtfully created "HD wallet" crypto code generating bad pubkeys.

I would strongly suggest that at a minimum to implement a redundant check of some sort.  Bitcoin core signs and verifies a message after generating each key.

When you import an address you can set a flag to force it to not rescan. So long as you're sure that you import it before any use was possible there is no need to rescan.