Red and blue are labels we assign to different radio frequency ranges that exist in Nature.
Yes, we know that now because we understand the physical nature of light. But we were able to use colors long before we understood that.
Radio frequencies can be measured and thus can be proven to exist in Nature.
They can be measured now, but we used them long before we knew how to measure them. As for us being able to prove they exist in nature, we used them validly for a very long time when the only way to prove they existed in nature was to point at something and say "Look! See the color?". We can do all the things for rights that we did for colors before we understood their physical nature. And we used colors validly.
How do you measure what or whom grants me the right not to have someone shine a million watt spotlight in my window at night and what or whom doesn't grant me the right to not have a flashlight brush across my window? You can't because those are abstract ideas that exist only in our minds and not in Nature.
People made that exact same argument about colors before we understood their physical nature. "If someone says the sky and grass are the same color, how can you prove them wrong? Colors exist only in the mind, so you can't use them."
We don't know how to measure rights yet, just as we once didn't know how to measure colors. But we can use them because we perceive them directly, just as we did with colors. If someone says "I believe I have a right to torture children for pleasure" or "The grass and the sky look the same color to me (under ordinary conditions)", all we can say is that they are either lying or somehow their perceptual mechanism is broken. It is impossible to convince a person that he does not see what he knows he does see.