I agree with Notabot. I am following it loosely, images don't work with some things (like fiction stories unless you draw them yourself) I don't want to penalize someone for that. I do look for grammar, spelling, etc and plagiarism of course. If it is a non-fiction, I look for backing information and links, but I don't pay much attention to how I feel about the article so much as if it is a good article or story. I don't think I should penalize the author just because it was something that didn't catch my fancy or that I wasn't interested in.
Is it worth editing the rating guide page and putting a summary of how you rate it (in the same vein as what's there I suppose), so other raters in the future can see it's really up to them how they rate it?
It does say it's up to the rater, but this round I found myself following the basic guide more than just doing what I did the first time I rated. My ratings tended to go more towards "Strong references and logical arguments" for opiniony/blog/non-fiction things, and images only if I felt it would have made it easier to read/understand the topic.
I go based on a mix, really. If it's something that should be factual, I expect proof of where the information came from. If it's an opinion or something, I don't. Images where needed are important as well, to me.
But, with that said, every article is different. I'm an authority source on what I write about; so for me to link to *other sources* isn't really possible; I'm the one people come to for information. Not the other way around. And I understand that there are a couple others here that may be in a similar situation. I try to judge the best I can.