Privacy concern would be that it would be obvious which testnet addresses and mainnet addresses belongs to each other, public keys would be the same.
That depends on the wallet... and the way it is deriving addresses... theoretically (
according to the registered coin types in SLIP0132), Bitcoin mainnet should be using the "coin" value of "0":
m/44'/
0'/0'
m/49'/
0'/0'
m/84'/
0'/0'
whereas... Bitcoin testnet should be using "coin" value of "1":
m/44'/
1'/0'
m/49'/
1'/0'
m/84'/
1'/0'
So, the account extended xpubs etc,
should be different for each network... even when they are generated from the same seed.