So , are people saying that i should UNCONNECT from connected sites to m.mask , then re connect when i need to use them ?
There is a great distinction between disconnecting some sites or revoking smart contract access.
To put up simply for security concerns, if you are dealing with established platforms like Aave, Uniswap, PancackeSwap, etc, you shouldn't worry too much. But if you are dealing with shady and unknown platforms or token or smart contracts, well this part should be better to avoid in the first place because if you already dealing with it, there is little contribution from disconnecting or revoking things.
For better references, I strongly suggest you read the information from Metamask: