Colored text is not the problem. The way you design your text is annoying because it unnecessarily screams for attention as if it demands to be read. Like for example, the text in this topic. Why would you feel the need to make it bold? Did you think that otherwise people would not read it?

Design your text in a non-spammy way and you won't hurt our eyes or annoy us.