How I see it, shouldn't you've more worried about your wallet being accessed through a vulnerability in the connected websites, that the other way around?
AFAIK, a vulnerability on either side can lead to a breach into the other; so if your wallet is accessed by a hacker they can gain access to connected sites. Also, if you link your wallet to a malicious site, it could lead to a breach in your wallet security.