...does this mean if wallet is accessed by hacker that they can then access connected sites ?...
If wallet itself is accessed, you are pretty much done for. Connected site or anything else becomes irrelevant.
However, no one can gain access just because you have wallet connected to some site.