Review Moderation & Trusted Users/Reviewers [DONE]

[Iceman[grrrr]]

I thought I was completely lost but now I know I'm ok! héhé

I second everything Elberet proposed except the 2 days period. Not everyone can login every 2 days! Even me am not always here on weekends so... Make it 5-7 days! Anyway, a review is never lost. And it could be shown as soon as someone accepted it, just give it a "Not yet determined" status until you have all the votes.

[Elberet]

Ah, reread the suggested voting mode, Iceman.

Quoting myself, with details adjusted according to other suggestions:

Voting modus:

• each trusted reviewer has 1 vote.
• an option [approve / reject] must have at least 5 votes to win.
• the voting is ended immediately if an option leads by more then 1:3 votes (e.g. 2:7, 1:5) or gets the absolute majority.
• the voting has a time limit of two days.
• if after the time limit, no option has at least 5 votes, the time limit is ignored until this condition is met.
• after the time limit, the win condition is modified and replaced with a simple majority condition. all voters that have not yet voted are assumed to abstain.

So this means:
Before the two days limit, the voting can end with 0:5 or 1:5 votes (approve has 5 votes and leads with more then 1:3), 2:7 or anything:10 (absolute majority has been reached, assuming there are 19 voters).
After the two days, the voting ends with 4:5, 5:6 or 3:9 (in each case, the voting did not end before because approve didn't lead with 1:3). The users who didn't vote yet are ignored.
If the current standings are so that no option has 5 votes (anything up to 4:4), the time limit is extended [ignored] until one option reaches 5 votes. In the case of 4:4, the next user who votes can decide the outcome all alone, since their vote would bring the standings to 5:4 or 4:5 which immediately ends the voting (since the time limit has past and simple majority is sufficient now). (For this reason, voters should not be shown the current standings until after they voted, and it should not be possible to change a vote!)

The time limit is freely chosen, and there's no reason for not extending it to 5 days or even a week. But keep in mind that this means that there'll be quite a large backlog of reviews that voters might have to go through...

[nwa]

you're talking about Review Moderators...
we (Gambit, me, r3quiem(although he was lost also for a moment there..) etc. were talking about Trusted Reviewers, they DON'T moderate any reviews, no approving, no deleting...

Code: Select all

public class Review Moderation & Trusted Users/Reviewers {
  public static void main(String[] args) {

     if (Review Moderator != Trusted Reviewer) {
         System.out.println("Correct!");
     {
     else {
         System.out.println("Bullshit!");
     {
  {
{

as we have already talked about the approving of reviews but it wasn't liked by everyone as ppl thought it would be unfair for others not able to post their reviews, there hasn't been much spam in AniDB since the review rating was implemented as there is a minimum character barrier and frankly I havn't seen any spam review, requiem says he has but he didn't link to those reviews so maybe he was just playing an ass as always

TRs also do not get any special recognition. The status of being a TR should always be a functional and never a honorary status.

meaning that others will be able to vote for even TR's reviews even though they are trusted? That is one of the problems we are trying to solve..I think...

Gambit!!! Say something!! you got the most reviews so you know for the best!!!

[Gambit]

Gambit!!! Say something!! you got the most reviews so you know for the best!!!

uh ... I`m lost for a moment. Gimme some time to think of a reply .. currently facing a deadline, so I`m not getting involved right now.

[edit]
I like Elberet`s idea, but the only thing here is that quite some people (kamenoko, DonGato, iirc) don`t approve with people who have to approve all reviews and thus act like 'gods'. So we decided not to moderate the reviews and let everyone post with a vote-button, after a lot of discussion.
That`s why we introduced the Trusted Reviewer©Elberet
These Trusted Reviewers ... well, I dunno anymore what they were, but they were supposed to have their reviews to be shown a slightly different from others.
I don`t know where they should be displayed, I don`t know how or what, I only know that they`re different from other reviewers. I don`t mean that they should be always on top (that`s bad for non-TR reviews), but perhaps (like Elberet said) in a different section (AniDB reviewer Reviews & Reviews, perhaps?) or just in a different color to distinguish them from the rest.

Furthermore, TRs don`t have any different rights than just being able to vote for new non-TR members based on the reviews they have made, as soon as they reach the stats which should make them TRs.

In my plans they`re also supposed to check if a reviewer is good enough to become a Trusted Reviewer - and I mean that they (chosen randomly) should check whether the new reviewer is able to write proper reviews or not.
[/edit]

[edit 2]
Goddammit, now I still replied
[/edit 2]

[Iceman[grrrr]]

Since there is clearly 2 points of view here, why not ask exp of what he thinks of all that ??

It could help clarify everything! héhé

[exp]

Since there is clearly 2 points of view here, why not ask exp of what he thinks of all that ??

hm, this thread is long, so i expect it to be a lot of work x_X
-> you still have lot's of time to discuss it before it gets implemented :P

It could help clarify everything! héhé

everything clear now? }:o)

BYe!
EXP

[Elberet]

but perhaps (like Elberet said) in a different section (AniDB reviewer Reviews & Reviews, perhaps?) or just in a different color to distinguish them from the rest.

Nani? When did I say that?

The problem with your implementation of Trusted Reviewers is that it essentially gives these users an advantage over others as their reviews are shown in a different color. This difference alone will entice kiddies to try and abuse the system, and they always find a way to do so, eventually.

Besides, this implementation does not help to cut down on the number of spam reviews with no real content or incredibly biased votes (all-10s or all-1s). To do so, a review moderation seems necessary. Now, I agree with you that moderation in general is a problematic thing, especially when it's about something like reviews where personal opinion is an important factor. And that's where my implementation of trusted reviewers comes in: A trusted reviewer is just a normal user. The fact that they are a trusted reviewer does not make them a moderator, it only makes them possible candidates for being randomly selected as a moderator for one review. Hence, all reviews will be approved by different users. The review moderators [voters] do not know who the review author is and the author does not know who the moderators [voters] are. The randomness and group factor involved in this process are supposed to prevent all cases of bias and favoritism.

If you want to avoid the term "moderation", you can easily think of this system as a "group vote among a subset of the regular userbase". A single user or even a small group of users do neither have the knowledge nor the power to influence the outcome of this voting process.

Edit: Between all the definitions, I forgot to make my point:
Trusted reviewers != review moderators. Period.
A review moderator would have privileges to edit/delete reviews at will. This is not the case for "my" trusted reviewers. In terms of SQL:
insert into review_voters (review_id, voter_id) select $rid, id from users where trusted_reviewer = 'true' order by rand() limit 19;
When a user gets selected this way, they become a moderator for this review until the review has been approved or denied. Once this vote is closed, the user has fulfilled that function and does not posess any further control over what happens to the review. Due to the number of votes that is required to approve of a review, personal opinion and bias are evened out, and what remains is hopefully a representative, objective decision. Idea: if a group of randomly selected people thinks a review is crap, chances are really, really, really good that the review is in fact crap.

[Gambit]

Now you only need people who don`t add their/other peoples name in the review, to ensure anonymousity, ne, requiem?

@Elberet, as I said, I didn`t have the time to do a thorough research on the posts you made, if I misunderstood anything, gomen.

[nwa]

sorry Elberet, I had trouble understanding you earlier but now I get it..
so a Trusted Reviewer is not a Review Moderator but can randomly become one for the given ammount of time of approving or disapproving the newly posted review.. Yea I like this idea, I support it

A trusted reviewer is just a normal user.

That is actually against mine and Gambit's opinion...
If a TR is just a normal user who can become a RM...then there is no point having a TR in the first place, as they only differ from others when they become the RM, but that can be done by being a RM in the first place.

This is how I see it, a TR is someone who got his status with the required stats(has passed the minimum ammount of reviews needed like 30, has passed the minimum ammount of user votes required like 200 and has passed the minimum total average rating like 5.5), after he becomes the TR, he does become an elite, I don't understand why some of you (DonGato, kamenoko, Elberet) hate that word and the concept... as quoting Gambit:

AR and AniDB has mods, mod = elite, mods are needed so elites are needed, so why is it any difference to have a review mod?

(note that I don't have the IRc log file here in uni so the structure of the sentence is altered quite a bit, but the idea is same )
so if we have RM's who are elite, we can also have TR's who are also elite, 1 has powers to decide on the fate of a new review, the other has the power to dodge that fate(including the immunity to yes/no votes), now in my eyes, TR's don't have to be the same ppl as RM's as that brings elitism to a whole different level, that is maybe called even monopolization although I wouldn't mind that as I would trust the persons who have the status just as I trust wahaha to take care of the CREQ's..why I trust them? because I know them and so do you, now for the ppl who don't know them, the ppl who you don't know you and you don't know them, would you be ready to let them become a TR? Yes, I would let them, cause it only does good. Yould you let them be a RM? No, I wouldn't as I don't know him and I "still" wouldn't trust him cause I simply hasn't talked to him...So would the "regular" reviewers who we don't know be against such elitism? No, simply cause either they don't know about it(cause they don't have to, also read the point in requiem's first reply here) or don't care, why would they care about that, these who care..why should they complain, we are not complaining that there are mods in the AR forum, we are not complaining that there are mods in AniDB are we?

Now the reason for a TR in my eyes, he is immune to any kind of criticism(can't vote yes/no to their reviews, but are TR's review's also being sent to a RM? Then there's a chance that one gets to do his own review although it is easily avoidable by coding a few lines), because, the reviews are obviously good, the reasons why these reviews mostly get voted no, is because of fanboys who's opinion differs from the reviewers opinion or just some lamers who think it's fun to just vote no etc. Now that is not right is it? read Gambit's review on Berserk for instance...now read it and tell me, is it worth the rating of 4.96? not really...I don't see a reason to vote a no for it...but there are ppl who have voted no...

Now before I talk about how I would see the reviews of TR's be positioned amongst the "regular" reviews, I want to know if there are other ppl who are supporting that idea or are thinking the same thing as I (although I think Gambit, requiem and theMug shouldn't have any objections at all)

P.S.: Keep in mind that the yes/no votes will NOT disappear for the regular reviewers even after the review is approved by the RM as we need the stats and the order of the reviews(which is on top etc.)

I hope I made my points clear now if they weren't before

[Gambit]

Just in case people think I said that out of the blue on IRC, I mean this:

A mod is in some way also an elite. They have more powers than regular users and they are supposed to keep things nice and clean. As long as it`s not competative (why would anyone become a mod if (s)he only wants to brag about it and abusing his/her powers? (s)he will probably be kicked out by the other mods because of that anyway), I don`t see how anyone would oppose an elite like this.
The only way I can imagine someone being against an elite, is when the elite will actually abuse the rights/status. Looking at the forums we`re hanging out, it`s more of a hassle than a status so I doubt the current 'elite' (i.e. mods of anidb, ar-forums) will abuse the rights of being an 'elite'. If so, please contact me

But as long as the 'elite' supports the community by keeping an eye on the quality and helping out, I don`t see a problem. If there is, please enlighten me

For the rest:

Now the reason for a TR in my eyes, he is immune to any kind of criticism(can't vote yes/no to their reviews, but are TR's review's also being sent to a RM?

Imo, yes. If you want to keep the reviews good and high quality, you also have to check on the quality of the trusted reviewer. Even though (s)he has proven his/her worth by becoming a trusted reviewer, it doesn`t mean that the person will never write a review which isn`t that good at all (rush job, bad day, etc). If you want to use TR`s for quality, you also have to check if those people live up to the standard. imho, that is.

Now ... we`re still talking about Review Moderators, I see? Is anyone able to explain to me what the exact difference is between a review moderator and a trusted reviewer and why we use a RM in the plans? Since I`ve kinda lost it between all the discussion posts along the way to this point.
I thought we already disposed the idea of a RM since that`s against the elite-building.

[Elberet]

so a Trusted Reviewer is not a Review Moderator but can randomly become one for the given ammount of time of approving or disapproving the newly posted review.. Yea I like this idea, I support it

Thanks, and yep, that sums it up. One important thing so: The group of randomly selected people becomes "the moderator", the individual is still relatively powerless, despite the ability to vote to approve or deny a review. (Afterall, it's one vote against 18 others.)

If a TR is just a normal user who can become a RM...then there is no point having a TR in the first place, as they only differ from others when they become the RM, but that can be done by being a RM in the first place.

Well, that's nomenclature again. For me, trusted reviewer is just a fancy word for a "can_be_selected_as_review_moderator" flag in the database. This flag is set or cleared automatically by the CGI (or maybe manually by an admin) during the n-hourly maintenance runs.

I don't understand why some of you (DonGato, kamenoko, Elberet) hate that word and the concept...

I don't really hate it, but if there is an elite, be it forum mods, AniDB mods, channel ops or even voices, some under-educated users (aka morons) will try to become a member of that elite for no other reason then soothing their inflated egos. As long as there is an alternative technical implementation that works without an elite usergroup, I prefer that.

1 has powers to decide on the fate of a new review

Ummm, no. The review moderators only have a fifth of that power, as at least five votes are required to approve or deny a review. That's why I prefer to call them trusted reviewers - they aren't moderators, they are normal users. However, they have shown that they know what's important about a good review, so the site trusts their opinion about a review.

TR's don't have to be the same ppl as RM's

Yup. Agreed. The question is even if we really need trusted reviewers. After all, one of the points of having trusted reviewers was to remove the public review approval votes from the site, since these are constantly being misused by users who think they should vote "no" just because the review didn't express their opinion.

Now the reason for a TR in my eyes, he is immune to any kind of criticism

Now that's a bad idea. I generally don't like the idea to give someone "ultimate" trust. The only user in a system who should be able to do things without *any* further checks from the operating system should be root. (In our case, we, the userbase, are the OS, and exp is root. )

P.S.: Keep in mind that the yes/no votes will NOT disappear for the regular reviewers even after the review is approved by the RM as we need the stats and the order of the reviews(which is on top etc.)

This is why I've said the thread has gone off topic. One of the earliest requests here was to get rid of these votes somehow, and basically all my suggestions here try to accomplish just that. Here's my chain of thought:

• We want to get rid of the votes.
• So how are reviews approved / denied?
• We need a few trusted users to do that.
• The mods already have too much to do.
• More mods means a higher chance that someone's mod account gets hacked into and abused.
• Review moderators could be accused of favoritism or bias, since reviews are a very delicate thing where personal opinion matters a lot.
• Consequence: We want to avoid giving a single user the permission to approve/deny a review.
• Consequence: If a single user can't do it, a group of user could do it.
• But the public approve/disapprove votes do not work well, that's why we want to get rid of them.
• Consequence: We need a vote among a number of trusted users.
• Now since we're talking about a large number of users with no power other then the privilege that the site asks them for their opinion from time to time, these users can be selected automatically, so neither exp nor any mods have to take care of that.
• If there are enough votes, bias and opinion are evened out.
• If every review is voted on by a different group of voters, a single reviewer can never complain that only his reviews are being disapproved or that everyone holds a personal grudge against him.

That's more or less how I developed those suggestions. Give or take a few corrections.

If you really want, trusted reviewers can always be added to this as well. This user status ("can_post_reviews_without_moderation" for the database) would allow the user to post reviews without having them queued for approval. If the review is shown in a special section, with a sepcial marker or in bold green letters on a sky-blue-pinkish-striped background... I couldn't care less about that. The important thing to me is to find a way to *replace* the yes/no voting once and for all.

[Gambit]

<nwa> Gambit: now remains the problem on the order of the reivews, but I guess the newest one will be on top...

I guess a nice solution for this is to keep the topmost review reserved for the latest addition, while below that one everything is sorted on height of the vote?

I don't really hate it, but if there is an elite, be it forum mods, AniDB mods, channel ops or even voices, some under-educated users (aka morons) will try to become a member of that elite for no other reason then soothing their inflated egos.

That`s why I don`t want to make it easy to get in such an elite. We already did that for the reviews, forcing users to type at least 500 chars (or something) which rules out most of the lamers already. The people who come out of it unharmed are supposed to be not lame at least. Next is to make sure who are able to write proper reviews, which is a little step higher again.

I never thought this thread was supposed to solve the problem of the voting-system by completely revoking the entire system and replacing it by a complete different system. That`s where I missed the point, I guess
In fact, I`m pleasantly surprised hearing (uh .. reading) it, and I think it`s not even a bad idea. But I think I`ve never said that about any suggestion of yours here, afaik

Taking into account how the reviews can be sorted, you could think about letting the TRs not approving/disapproving, but instead giving them a rating like: very bad, bad, decent, good, very good. This way you can sort the review-list on average rating by TR. Of course the first 2 are similar to disapprove and the latter three are approvals.

* Gambit wanders off, wondering why he never understood the part about losing the votes...

[Elberet]

In fact, I`m pleasantly surprised hearing (uh .. reading) it, and I think it`s not even a bad idea. But I think I`ve never said that about any suggestion of yours here, afaik

Thanks.

Taking into account how the reviews can be sorted, you could think about letting the TRs not approving/disapproving, but instead giving them a rating like: very bad, bad, decent, good, very good. This way you can sort the review-list on average rating by TR. Of course the first 2 are similar to disapprove and the latter three are approvals.

Let me quote myself... again.

Since all reviews that are actually shown there [on the reviews page] have already been approved, the three groups "Approved", "Unrated" and "Disapproved" are removed. The order in which reviews are shown could either be determined by date (latest review first), or by a secondary rating determined during the closed approval vote. "order by `date`desc" is of course the easiest method to implement, but in case exp turns out to be a coding masochist:

• Option 1: Determine the rating by the ratio of disapprove/approve votes during the closed vote. The rating is shown using the same icons that are used for the file quality:
  • "very good": disapproved / approved == 0 (no disapprove-votes at all)
    "good": disapproved / approved <= 0.2 (e.g. a 1:5)
    "medium": disapproved / approved <= 0.4 (e.g. 2:5)
    "low": disapproved / approved <= 0.6 (e.g. 6:10; 3:5 is not possible [unless after the timeout])
    "very low": disapproved / approved <= 0.8 (e.g. 8:10 or 8:11 if everyone voted)
• Option 2: Upon approving a review, let the voters also submit a rating for the review. Possible ratings are 1-5 (very low -> very good) and the average determines the final rating *if* the review is approved. If the review is disapproved, ratings are ignored completely as the review will never be shown anywhere.

[nwa]

Loose the whole voting system?
ah so ka...yea..that would make TR's useless in my version, damn why didn't I get it sooner, Elberet, you have to be more clearer about yuor ideas

Ummm, no. The review moderators only have a fifth of that power, as at least five votes are required to approve or deny a review.

where will you get that many TR's?
if the mods are left out...then there would be about a max of 5 ppl as TR's or RM's...

[Gambit]

@Elberet

uh ... I`m lost for a moment. Gimme some time to think of a reply .. currently facing a deadline, so I`m not getting involved right now.

Which means that I did read everything, but didn`t get it all due to the stress. And actually I still don`t have time to read it all

where will you get that many TR's?
if the mods are left out...then there would be about a max of 5 ppl as TR's or RM's...

In his original idea he said that people who have written at least 5 reviews with an average of 5.5 or higher will be TRs. Then there will be plenty