I'm not sure about the “right to refuse”. If you refuse to give service to someone, for example, because they're black, that's discrimination and a crime in certain jurisdictions (it should be everywhere).