If you're transferring your coins to a native SegWit address, such transaction would be invalid on the BCH and BSV networks because they don't have SegWit, so you don't need to worry about it. And other forks should have implemented replay protection to prevent such problems. I have many empty addresses that still hold the forked coins, and there wasn't any problems with replays.

Just make sure to prioritize the safety of your bitcoins first - never put a private key that still has Bitcoin balance in any altcoin software.