Review Moderation & Trusted Users/Reviewers [DONE]

[nwa]

revoke votes has been implemented, that was the original goal yeah..
nice

now about becoming a trusted reviewer..so if the whole userbase should decide on that, then how would we do that? put up a poll with Yes and No?

[Elberet]

Not the whole userbase...

The first batch of TRs are selected by the current review approval ratings. Any user who has submitted a certain number of reviews, has a certain number of votes on his reviews and has a certain average approval rating would become a TR. Once these "first generation TRs" are there, new TRs are voted on by the existing ones. When the CGI detecs that a user satisfies the second-generation TR requirements, a certain number of TRs are randomly selected and receive notification popups informing them of the vote. The vote on the new TR takes place in the CGI as well, so the outcome of the vote becomes effective immediately without requiring further administrative work.

[Iceman[grrrr]]

Even better than that, since the first generation of TR already voted for the candidates to the 2nd gen TR, they don't really have to vote again. If they voted to a certain % (make it high enough) then the candidate becomes a TR and his votes will count towards the next TR.

This saves the current TR from rereading the candidate's reviews to remember what he wrote!

What do you think ?? (Ok it would be hell to code...)

[Elberet]

Hm yeah, that would make the whole system less verbose. And besides, I don't really like the idea that new TRs are being voted on by the old TRs... :/

[Gambit]

Hm yeah, that would make the whole system less verbose. And besides, I don't really like the idea that new TRs are being voted on by the old TRs... :/

Well, you can see it as a last resort for people to check on them. They will automatically get accepted as long as there aren`t at least 2 TR`ers voting 'no' for adding that to-be-reviewer to the TR`s. I don`t see how often this will happen, but I doubt there will be much cases where there are people voting 'no' for a reviewer.
It`s definitely not that all TR`ers will have to check into and approve the new reviewer to become a TR. But if that reviewer is just a bad reviewer who gets high ratings because he pleases the fanboys instead of writing good reviews, I don`t see why it`s so wrong to let some TR`ers put a stop on him to become a TR`er.

[requiem]

I see a lot of code being implimented involving this random selection process of TRs...not only that, what if a first-generation TR doesn't appear for a while?

Although I can see how this may solve the problem of finding new reviewers, we are taking the risk on the original TRs already being there. I agree that it's a good idea, but just trying to help make it failproof. Also, if the reviewer is a fanboy reviewer, technically he is appeasing to the people and telling them what they want to hear. I would imagine as long as it's describing the story, characters, animation, value, and enjoyment - it doesn't matter what opinion they give.

However, if it comes to ratings being at ten, spoiling, spamming, and using a lot of bullshit to make your review long:

I suggest a "Fanfreak Limit." This can be a certain limit that must be abided by, or else the option of becoming a Trusted Reviewer is not possible. Not only will this lighten the workload on code development, but it may prove as effective as have TRs induct more TRs.

--

• One 10/10 Review MAX - of course, I am a victim of this 10/10 desire...however I would imagine a 10/10 review doesn't really hurt the overall review bias that the reviewer gets if they give that rating to one specific anime. However, if they keep doing this, then the system should deny them the ability becoming a TR; unless they either delete or edit the score on their incriminating reviews.
  
  Bullshit Awards / 3 warnings - Since the person isn't a TR yet, they will still be -part of the "vote yes or no" system. If there can be a "Contact TR/mod" link that will message a random TR/mod as to explain what sort of "bullshitting" or "spamming" a reviewer is doing, like say:
  
  I just love Love Hina! It is so good! I watched it 4 times this week!
  
  By the way, Love Hina just owns!!! If only more anime were like this that make me happy sad crying laughing and falling outta my chair
  .
  .
  by the way, did you know in inuyasha kagome shoots arrows at people? i know it has nothing to do with love hina but this is cool anyways.
  .
  .
  k im gonna go watch it so
  .love hina good
  .love hina great
  .love hina make you
  .love to hate
  bye!
  
  
  A rather exaggerated example, but I've seen it before...a fanboy who writes a long review filled with lots of marks and repetitions to sound creative and actually trying; this can easily be picked up if there is a PM to a TR, and thus a judgement should be quickly made, without necessarily permanently damaging the reviewers status. The TR/mod can then give the reviewer a "bullshit award point." When they get three of these, they cannot become a TR until they delete such reviews. I would imagine that TR be mature and honest, so there shouldn't be too much argument as to why they can determine the quality of a review.

--

Anyways, once somebody has violated one of the above standards, they are unable to become a TR until they resolve such conflict. Then, once they are finished, they become a Trusted Reviewer (verifying). The next review they write will verify their status (so it means that they confirm they know they are becoming a TR), and thus, become Trusted Reviewer.

I wrote this relatively quickly as to put my thoughts together in the late evening. I might have typoed or made a mistake, if anything doesn't make sense...I'll make edits.

~requiem

[Iceman[grrrr]]

well, I'm not really sure about that...

if the reviewer just has to delete the bad reviews to have a blank history, it's kinda easy... someone who submits a shitty review, even if it is only twice, should not become a TR...

and the main idea was having TR to check all reviews before they are shown! there won't be any yes/no votes anymore since that would be the TR job...

I think you have to clean your idea a little!

[nwa]

Iceman[grrrr]:
you got it kinda wrong...

the yes/no votes reamain..how would we able to recognize a potential Trusted Reviewer without his ratings?
and a Trusted Reviewer won't be checking on other ppls reviews, the point of the Trusted Reviewer is to be able to write good reviews and stay on top with them as he is immune to the yes/no voting
as I already said, so that Gambit's Berserk review could remain on top despite the large number of fanboys who think that Gats 0wns j00 and who would like to give a NO for the reivew..that's the whole point of it

requiem went kinda off the track also..actually I think he should be the first to get the "bullshit award point"

[Iceman[grrrr]]

and a Trusted Reviewer won't be checking on other ppls reviews

well you said it yourself back then you know!!!

my support went and still goes to Elberet's idea for having a mod(s) to check the reviews before made public

From what I have read, I thought TR were like mods for reviews, they delete bad reviews, approve good one, and ask the creator for changes if it is good but has something off... And that's why we don't really have to know how new TR are created because we don't need a lot! 10-20% of all reviwers should be TR, not more...

[nwa]

o..well I didn't see Review Mod's and Trusted Reviewers as the same thing but now that you mention it...


but that brings back the old problems that kamenoko mentioned

[Gambit]

From what I have read, I thought TR were like mods for reviews, they delete bad reviews, approve good one, and ask the creator for changes if it is good but has something off... And that's why we don't really have to know how new TR are created because we don't need a lot! 10-20% of all reviwers should be TR, not more...

Nope, a TR is a Trusted Reviewer. A TR is someone who has earned some respect by making a lot of good reviews. The only thing they get in return is that they get the rating Trusted Reviewer. They certainly won`t get the ability to delete other reviews or anything.
I`m unsure how to see nwa`s idea ... I didn`t think a review of a TR stays on top, since then you`ll have to keep all reviews made by TRs on top. I rather thought that a TR gets a little mark beside the name to indicate that this person has earned respect by writing a lot of good reviews. Imagine some series where both requiem, nwa, BMan and myself have written a review for - then there`ll be 3 pages (considering our long reviews ) and they all stay on top? Kinda depressing for others So I don`t agree with that. I`d rather see a sort of label to indicate that someone has made more good reviews, even though he might not be the top rated reviewer for that anime.

I see a lot of code being implimented involving this random selection process of TRs...not only that, what if a first-generation TR doesn't appear for a while?

No sweat. As Elberet already said, there will be a random selection between TRs. Although there might not be enough TRs at the moment, there will be. It`ll only be a matter of adjusting a little number in the code eventually.
About your Bullshit Points (BP), it could help ... this way you only need 1 person to shoot some BP to deny someone becoming a TR. I think there should be a little check on that, so people won`t BP someone who they don`t like. I like Elberets ideas of being random, but that doesn`t work with BPs.

[nwa]

I`m unsure how to see nwa`s idea ... I didn`t think a review of a TR stays on top, since then you`ll have to keep all reviews made by TRs on top. I rather thought that a TR gets a little mark beside the name to indicate that this person has earned respect by writing a lot of good reviews.

hmm..a TR gets a little mark..but he is also immune to yes/no votings as they are not needed for TR's right? I mean the little mark only wouldn't change anything, ppl can still vote no maliciously for your Berserk review even tho it's good...
so if we all agree that TR's can't be rated cause they are already that "good", then it means that they will be moved automatically into the Approved Reviews side...they will be on the top anyway..until someone writes a new review and it gets 3 user votes to get to the Approved Reviews section making that review be on the top..but that also means that TR's cannot move back to top, they can only move back if someone gets approved again..which is kinda strage then, a "normal" reviewer stays on top while a TR, while still being in the Approved Reviews section, remains on the bottom...

[requiem]

so if we all agree that TR's can't be rated cause they are already that "good", then it means that they will be moved automatically into the Approved Reviews side...they will be on the top anyway..until someone writes a new review and it gets 3 user votes to get to the Approved Reviews section making that review be on the top..but that also means that TR's cannot move back to top, they can only move back if someone gets approved again..which is kinda strage then, a "normal" reviewer stays on top while a TR, while still being in the Approved Reviews section, remains on the bottom...

Hmm, nwa - that does raise a good point altogether...it kinda makes me wonder: if the TR system is implemented, to avoid "favoritism" and such that would keep others from becoming TRs, where exactly should these TR reviews be placed?

I honestly wonder about all this now. By discriminating between the TR's reviews and the regular reviews, we give all visitors to anidb a set of decent, well-written reviews - but we also close off the opportunities of newer reviewers to be selected. However, if we keep the rating system that is now in control, many of the honest reviewers are being pissed on by fanfreaks and otaku-wannabes. With a double-edged sword like this, I think there is less need for priority and more for opportunity.

It recently occured to me...how about we put the trusted reviewers in a section at the bottom or right hand side of the page? The "ranked" reviews will be at the top of the page, in their own portion, and viewers can vote for the reviews as what is already in place...

...however there can be a highlighted or boxed off portion at the right hand side (or bottom, but that can make problems) specifically known as a Trusted Reviews section. The trusted reviewers will have either a different style font, or have it an inverted-style color display, such as light-gray on a blackish-grey background. This way, the Trusted Reviews can demand more attention to the eye, but they won't inhibit the reading of the regular reviews (and long reviews won't be a problem).

The only problem is that the window might get too cramped, especially if someone is using 800x600 resolution (and even then - the windows at that res are too small for anidb's appearance). However, if the TRs are at the bottom of the page, and there is a long list of "ranked reviews" above them, then we will just have to hope readers remember to scroll down.

--

Anyways, I see that the Bullshit Points didn't really make sense - I'll just go along with Elberet's idea; I just hope the workload in coding won't be too exponential.

[Elberet]

Now this thread has seriously gone off topic...

The original purpose was to build a system that

• prevents spam-reviews by not accepting them as reviews in the first place.
• eliminates the issues with approval ratings, since the common voters are usually biased and vote yes/no depending on their own opinion about the anime rather then the usefulness of the review.
• does not put more stress on the existing AniDB-moderators.

To fulfill these requirements, I suggested that reviews be moderated before they are made public. During this moderation phase, a number of randomly selected users who are part of a certain subset of the normal userbase known as trusted reviewers, will be shown the new review anonymously (without the name of the user who submitted that review, to prevent biased votes) and the outcome of this vote determines if the review is made public or not.

Once again: a newly submitted review is not visible to anyone except the few users who were randomly selected to vote whether the review should be approved or not.

• Trusted reviewers do not have any special privileges whatsoever. The one and only difference between a normal user and a trusted reviewer is that the system considers a TR as a potential voter when it comes to selecting users to approve a new review.
• Trusted reviewers are not exempt from review moderation. They can not vote on their own reviews.
• TRs also do not get any special recognition. The status of being a TR should always be a functional and never a honorary status.

Furthermore:

• The difference between first-generation and second-generation trusted reviewers is *only* in how the site associated them with this status. Once selected, all TRs are equal in status and function, regardless of how they were selected.
• The whole thing about generations was only brought up because Gambit suggested that new trusted reviewers should be voted on by the existing reviewers. However, I oppose this suggestion. A purely statistical approach sounds more useful to me: Once a user has submitted 20 reviews that were approved and appeared in AniDB, it is to be assumed that this user knows how to write a good review and knows how to properly vote if a new review is ok or not.

Lastly, what happens if a user who has the trusted reviewer status goes inactive? I'd suggest this:

• Add a new perference to the userprofile which is only visible for users who are considered trusted reviewers.
• This preference allows a user to select whether they want to be bothered by voting on new reviews - after all, it should be possible for someone to post new reviews without having to care about someone else's.
• All review approval votes have a time limit of 2 days. If the user does not view the review within that time, the preference flag is automatically set and he will not be considered as a voter anymore unless he visits his profile and resets it. An automatic private message is generated to inform the user of this change.
• If a user is selected to vote on a review three times, views the review every time but does not actually vote on these reviews, the same preference flag is set automatically and an automatic private message is generated informing the user of this.

[Elberet]

Ah, by the way:

The review page for an anime would be restructured as well, of course - at least if you decide to follow my suggestions. Since all reviews that are actually shown there have already been approved, the three groups "Approved", "Unrated" and "Disapproved" are removed. The order in which reviews are shown could either be determined by date (latest review first), or by a secondary rating determined during the closed approval vote. "order by `date`desc" is of course the easiest method to implement, but in case exp turns out to be a coding masochist:

• Option 1: Determine the rating by the ratio of disapprove/approve votes during the closed vote. The rating is shown using the same icons that are used for the file quality:
  • "very good": disapproved / approved == 0 (no disapprove-votes at all)
    "good": disapproved / approved <= 0.2 (e.g. a 1:5)
    "medium": disapproved / approved <= 0.4 (e.g. 2:5)
    "low": disapproved / approved <= 0.6 (e.g. 6:10; 3:5 is not possible)
    "very low": disapproved / approved <= 0.8 (e.g. 8:10 or 8:11 if everyone voted)
• Option 2: Upon approving a review, let the voters also submit a rating for the review. Possible ratings are 1-5 (very low -> very good) and the average determines the final rating *if* the review is approved. If the review is disapproved, ratings are ignored completely as the review will never be shown anywhere.

Other random stuff:

• Voters should be given a chance to add feedback when disapproving of a review. This is done anonymously: The voter doesn't know who he's voting on, and the person reading the feedback doesn't know who it's coming from. (However, do keep track of who sent the feedback and add a feauture that allows users to report offensive feedback.)
• When a review is disapproved, don't just delete it. Give the user who wrote the review a chance to edit and resubmit it. A nice place for this would be the myreviews page: Extend it to show approval/disapproval status, a link to the voters' feedback and a form to edit the review if it was disapproved.