Quote from: theymos on May 12, 2018, 04:49:15 PM

<...>
I am particularly interested in identifying people who should be made merit sources, or existing merit sources who should be given a higher monthly limit. When I originally created the system, I thought it'd be fairly easy to look through the stats and figure out who would make good sources, but it's been more difficult than I expected. Is there any additional data I can provide that would be useful?

In order to help identify merit sources, this kind of analysis is a step towards it, but there are a few things to consider in my opinion:

1. The first need is to identify merit awarding lacking areas. A network analysis of already existing sMerit awarding TXs, being it either from o global perspective or a local one as was my approach, is a path to identifying merit abuse and possibly alt account ultimately. Nevertheless, it is based on user to user relations as a primal driver, where posts would need to be the focus to identify Merit awarding lacks.

My approach would be to analyse boards, at the very least at section/subsection level. For each of these we have to assert if there is a need for additional merit sources. This analysis is threefold:
a) Determination of the current amount of sMerit and Users awarded in the section/subsection.
b) Determination of the number of posts in the section/subsection.
c) Quantification of the quality of the posts being posted in the section/subsection.

These are not simple isolated dimensions, since they should be measured in a certain timeframe. The overall measure of sMerit awarded is fine for a general overview, but in order to asses if a new merit source is needed we should limit it to a given timeframe.
Just as to determine the stock of raw material needed for production we need to see it in a narrow framework of time and not considering all the historical data for day to day operations.

a is covered and can be traced I guess internally. We got a pretty good glimpse of it last week ( Forum Metrics - Section/subsection sMerit breakdown ->section 5), where we got to see a weekly breakdown of awarded sMerit per section/subsection.
We should forget the first two or three weeks of sMerit fiesta and focus on the rest. That is the real week to week sMerit awarding that takes place in a given section/subsection. Lets consider sMerit the boxes to contain the produced goods.

b is the actual production (number of posts in the section/subsection). We know that posts are awarded retrospectively, but the vast majority are awarded within a week or two timeframe at most from the moment it was created. Alas, we don´t have those figures. Perhaps the number of distinct associated users involved would also be useful.
b is not considered in a network approach sMerited TXs, since the view is biased to merited posts specifically, not total number of posts.

c is the human quality control of production that must be performed upon b. The idea is to give us a quantitate measure of the qualitive side to the posts. This is the toughest one to get right, and should yield as much objective criteria as possible.

Lets take for example the Spanish Local Board. It was awarded 35 sMerits on the week of the 23/07/2018. Let´s say that there were 350 posts created that week , and that the overall perception is that those 500 posts had an overall quality translated into a 15/100 (that is, the perception is that 15% should be awarded sMerits).
The above scenario would play out to say that roughly:
500 * 15% = 75 sMerits is the minimum expected awarding for this weeks batch. Since only 35 were given out, it would mean that there is a deficit of another 40 sMerits to apply to normalize reality to expectancy, thus a source would be required.

I know, the above is rather simply played out, and the hard part is getting the c right since it is key in the dimensioning process. Theres also the fact that some posts should be awarded multiple sMerits, but the principle is still bases on these three factors. It can be further complicated by bringing the number of posters into the equation too.
The above can be set with a weekly timeframe in mind, or something larger like a monthly window.

Now whats left is determining the sorce/sources. That is where we would postulate as candidates users that award most in the given section/subsection, combined with a wider spread of awarding history over multiple posts and/or users. That is, better to have as a source someone who awards few sMerits to many users that a lot to a few users.
Divide the sMerit deficit by the average you want to give to a source, and you get the number of sourced to have in a given section/subsection.

Its not perfect as finer details need to be outlined, but thats how I would approach it and build up from there.

Theres one more thing that buggers me which is, regardless of it all, what is a good overall sMerit circulation that is needed to keep the system healthy, working, and having users believe in it?.
Too much sMerit circulation would render it useless, as it would defeat the purpose, but having a very restricted amount may drown the system and its credibility through inanition.