Side channel attacks shouldn't be your main concern, secp256k1 libraries has mitigations against most of that so you'll be mostly safe. Problem would be with malware or some $5 wrench attack.

Privacy concern would be that it would be obvious which testnet addresses and mainnet addresses belongs to each other, public keys would be the same. I wouldn't do so though, leads to unnecessary confusion as well and perhaps leads to unnecessary risks depending on what you're doing with your testnet.