You can block others from adding you to groups by changing your privacy settings. Whenever someone PMs you, just check to see which of the groups you have in common before responding to them.