As far as I can see, the only security issue would be if your Blockchain.info wallet was compromised, one could see what address you own.

And still, the only "security" that is lost is anonymity.