I know enough to know that I left all those names removed intact because you may be trying to scrub a negative off someone and that is why I am at least suspicious since I now removed as many as I did has someone been lift from negative to positive?
What assurance do I have that my actions did not help someone today by pruning the list?
The solution to that is somewhat simple. Go through the sent ratings of the user you're going to remove, and back up the negative ratings in case that they are valid. Shouldn't be a problem with the claimed (by OP) inactivity/lack of activity of said users.