Review Moderation & Trusted Users/Reviewers [DONE]

[Elberet]

where will you get that many TR's?
if the mods are left out...then there would be about a max of 5 ppl as TR's or RM's...

In his original idea he said that people who have written at least 5 reviews with an average of 5.5 or higher will be TRs. Then there will be plenty

Exactly. Due to the random selection and the relatively large number of voters, it doesn't matter (much) if one out of ten users is actually not a very good reviewer. However, just to be on the safe side, I'd suggest that - if this system ever gets implemented -, exp should test the numbers by running a script on the DB that spits out the names of everyone who would be selected and making sure there are enough users and not too many "bad apples".

[Gambit]

Currently the list of reviewers is a bit short though.

• 118 gambit
  84 themug
  50 requiem
  29 nwa
  25 petriw
  23 aranaxon
  22 mrsatan
  18 lion
  11 lamerde
  11 samiloop

This is the top 10 of reviewers at this moment. Which makes pretty clear that there won`t be a really good randomisation of 19 people if you want to do it that way.
I already suggested that you`d better use the minimum of 20 reviews with an average of 5.5 - or 10 reviews with an average of 6. But even in that case you don`t get very far. Especially since the people in this list don`t all have a rating over 5.5 average.

My new suggestion is to implement that anyway (10 reviews, rating of 6 or 20 reviews with average 5.5), with the randomizer, and that the number of TRs gradually grows. It might seem a bit unfair at the moment, with only 5-7 people as TR, but keep in mind that it`s just a new system. Hell, even the review-option isn`t that old yet, so gradually there will be more people writing reviews, and there will be more people becoming TR. I expect at least 10 people somewhere beginning next year After all, not everyone likes to review, but I`ve seen plenty of people who wrote 2-3 reviews in a row (on a whim?) and then stopped for the time being, just to continue it after a few weeks again.

[nwa]

hai hai I'm supporting Gambit the Review Machine

as...mrsatan, has way lower rating than 5.5, aranaxon is dead(read "missing") and he has a rating below 5.5 anyway, lamer_de has lame reviews, so only the first 5, lion and samiloop would pass from top 10

[Elberet]

and that the number of TRs gradually grows.

Sure, that should not be a problem. The only implication this has would be that everyone has to vote on every review every time, but since the requirements are rather tough in your suggestion, fairness would be preserved. Alternatively, it's always possible to consider the mods as possible voters as well. And another possible way would be to adjust the requirements so that there are 38 trusted reviewers. If more people satisfy these requirements, there aren't more TRs but the requirements are moved up so that there are still only 38 TRs. Once we've reached the desired reqs, the number of reviewers is allowed to grow.

After all, not everyone likes to review, but I`ve seen plenty of people who wrote 2-3 reviews in a row (on a whim?) and then stopped for the time being, just to continue it after a few weeks again.

Yeah, I know what you mean. I myself, for example, only consider writing a review when I have just finished the series (give a week to let it settle down) and have some serious praise or rants to dish out. That might not be the best way to write them, tho.

[kamenoko]

I've been catching up on this thread, and it seems like I spurred on a bit of a discussion. I believe that is a good thing. What I have yet to see (I haven't completely caught up yet) is any suggestion that addresses the heart of my original argument: That the review section should be classless.

That is not to say classless in the sense that it sucks, but classless in the sense that one person doesn't hold power over another. This is the way it was before the review system was put in place, and I saw no problems with it.

Perhaps this review system was put in place because of the fiasco that is the animeNFO, and how bad it can get when there are 30 - 40 reviews per anime and 25 - 35 of them suck. We're not at that point yet, I doubt there is an anime out there with more than 10, and that's easily manageable on an individual basis.

There is something else wrong with a peer review system in any form, ego, and ego kills. The moment we allow ego to dertermine what we see and what we don't, we're done. I've seen it happen too many times in social groups to think we're immune, we're not. The second we have someone who is absolutely abhorrent at everything smelling of review yet thinking they're God's gift to anime, we have a cancer on our hands that's hard to dig out.

But what I realize now is that I identified the problem without offering a solution, and if someone has already suggested it in this thread I apologize and give you full credit. Let users ignore reviewers who suck, or reviews that suck. It's a simple, elegant solution that puts the power back into an individuals hands in an effective and positive way. If the user doesn't want to see it, and doesn't see it, we have a happy user who has godlike power over himself and no other.

Thank you for your time.

[Gambit]

But what I realize now is that I identified the problem without offering a solution, and if someone has already suggested it in this thread I apologize and give you full credit. Let users ignore reviewers who suck, or reviews that suck.

The problem I see there is that, when there are more than 3 reviews, it takes some time to read it all, especially since we have the condition of 500 characters to dig through. If you don`t have that, you`ll get a review-list like animenfo, 25 pages with reviews smaller than a rats ass and useless.
Now, imagine you have to dig through 25 pages with reviews of 500 characters, where half of them aren`t even close to useful, but are written only to promote the anime. At AnimeNFO, you could skim through them, at AniDB you can`t because they`re most of the time too long to skim through.

Although I agree with you that we at AniDB don`t have 25 pages, even 3 reviews is already a lot to read. It`s a pity if you read one and think of it as crap. That`s why the voting-system got accepted, to ensure that the good reviews will actually have a chance to stay on top instead of disappearing to a 2nd page or so.
Now it has been proven that that doesn`t work, so we have to think of another way. Of course you can say that you can let the user decide for himself whether a review is good or not, but as soon as you have to dig through 2 pages with 5 reviews of 500 words each, you`ll realize that that`s not worth the time and you`ll probably watch or don`t watch it without paying attention to the reviews. My goal is to make a system where the first (or first 3) are good enough reviews to make it easy for the reader to decide whether to watch or not watch the anime.
You can call that forcing it on people, but hey - they`re not forced to read a review, but if they want to read a review for it, it`s best to help them a bit and bring out the best reviews for them.

At least, that`s my vision on it.

If you think a review is just to view your opinion, go visit a forum, or a chat and voice your opinion. A review is to help people determine whether they want to watch it or not, which means it should be useful. Of course your opinion matters, but you should also remember that some people don`t give a shit about your opinion. And then your review should still be of some use to them.

If there are so many crappy reviews/spam like AnimeNFO, people don`t even bother to read them anymore. It`s more like a playground to bash/praise anime, and that`s not helping anyone. If you want to let users do that in AniDB as well, then AniDB won`t be much different from AnimeNFO in a while. That`s definitely not what you want to have. So, although I like the idea of everyone being able to post something, you do have to maintain a certain amount of quality, if you want to make the review-section useful, instead of friendly.

But that`s just again my point of view.

[Elberet]

The second we have someone who is absolutely abhorrent at everything smelling of review yet thinking they're God's gift to anime, we have a cancer on our hands that's hard to dig out.

Yup, that's why I don't want any single user to receive privileges over any other user, and that's why we have all the talk about voting and selecting and so on and on. Where individuals cannot be trusted, a group of anonymous individuals can. Knowing your peers and knowing what you're doing is important for someone who wants to abuse a system. Take away that knowledge, and abuse is impossible.

Let users ignore reviewers who suck, or reviews that suck. It's a simple, elegant solution that puts the power back into an individuals hands in an effective and positive way.

That's a good idea as well, but there are imo three problems:

First, a user would have to eventually ignore many reviews and many review authors. If an author learns from his mistakes and starts writing good reviews, the users who've read the crap review will not see the good reviews either. As a result, reviewers are never forgiven their early mistakes.

Second, anonymous users (read: guests who aren't logged in) can not make use of that system. They will see all the spam reviews, all the crap, all the biased junk and all-10 votes.

And third, currently, an anime has two rating values: A number derived from the normal rating votes, and a number derived from the average ratings in reviews. This average reviews rating gets fully influenced by all the bad reviews out there, and even if users can hide the bad reviews from display, the rating will remain false. The only way to correct that specific problem is IMO to moderate reviews in some way or another. And with "some way or another", I mean a system that puts a varying group of random (but qualified) users in charge instead of single people.

[Iceman[grrrr]]

Ummm, no. The review moderators only have a fifth of that power, as at least five votes are required to approve or deny a review.

where will you get that many TR's?
if the mods are left out...then there would be about a max of 5 ppl as TR's or RM's...

Well Elberet sugested that all TR are candidates to become RM... say there are around 50 users in those... (well maybe less I don't really know...) They must satisfy certain criterias and are automatically set by the CGI during one of the system check. Then, every time a new review is posted, the system chooses 19 users in these 50 and they become RM. These 19 votes on the review... The point is they are always a different set of voters thus preventing all consequences of having the same people voting...

[Elberet]

Iceman, the problem we identified here before is that the criteria that were suggested before - 10 reviews, min average approval rating of 5.5 - would only give us 7 instead of 50 candidates for the approval votes.

How about lowering the requirements? Say, 5 reviews at an average approval rating of 5.5 ?

[nwa]

nope, it's fine as it is, actually I don't really like that the RM's would be such a large group...10 ppl would actually be quite enough...choose randomly 5 from 10..that isn't so bad..oh well

now if we decide to take 19 ( ) RM's...then how are we able to identify them if the review vote system will cease to exist..wait til we got 19 users and then take the votes down?
or lower the requirements as Elberet suggested? 5 reviews..that way we don't even know if the person that has writen only 5 reviews is even dedicated, as active in AniDB..it's risky to take ppl you don't know into such a group..you want to prevent the birth of elites who use their power in a wrong way? those ppl that you don't know may do that, the ppl you do know wouldn't...

[Iceman[grrrr]]

When the RM's votes are implemented, take that to identify more people!

like 5 approved reviews with an approval of 80%

[kamenoko]

[Yup, that's why I don't want any single user to receive privileges over any other user, and that's why we have all the talk about voting and selecting and so on and on. Where individuals cannot be trusted, a group of anonymous individuals can. Knowing your peers and knowing what you're doing is important for someone who wants to abuse a system. Take away that knowledge, and abuse is impossible.

There's a few problems in that system.

1. The system depends on the previous peer review system, which many have thought is flawed. The continuation of that system is also necessary to make the new system work (how else will the pool of trusted reviewers grow?)

2. The system will create two classes, those who can approve reviews, and those who can't. Philisophically I feel this cannot work. I don't know what the rest of you think, but I don't want an elite class of users.

3. What happens to the multitude of reviews that are already freely available, good and bad alike. Will these eventually be held to the same review system that all new reviews will be, and how will the logistics of that work?

That's a good idea as well, but there are imo three problems:

First, a user would have to eventually ignore many reviews and many review authors. If an author learns from his mistakes and starts writing good reviews, the users who've read the crap review will not see the good reviews either. As a result, reviewers are never forgiven their early mistakes.

Make the bans variable length, say for a month, or a certain amount of reviews. Then if this person eventually get's their act together, their reviews will not be ignored.

Second, anonymous users (read: guests who aren't logged in) can not make use of that system. They will see all the spam reviews, all the crap, all the biased junk and all-10 votes.

Seeing as how the primary purpose of the anidb is I believe to build and maintain one's anime list, and you have to be a registered member to do that, protecting the interests of anonymous users, or subjecting them to our ideal of a good review without their imput should not be in our purview.

And third, currently, an anime has two rating values: A number derived from the normal rating votes, and a number derived from the average ratings in reviews. This average reviews rating gets fully influenced by all the bad reviews out there, and even if users can hide the bad reviews from display, the rating will remain false. The only way to correct that specific problem is IMO to moderate reviews in some way or another. And with "some way or another", I mean a system that puts a varying group of random (but qualified) users in charge instead of single people.

I think the rating based on the review system is too easily skewed to be a worthwhile number, and nor shall it be based on the fact that we have to choose 6 criteria as opposed to choosing a basic overall review number is distressing. Especially when two of the criteria (Enjoyment and Value) are so close toghether it's like adding twice the weight to one criteria. But that's a story for another topic.

Whatever happens will happen. I merely believe that the answer to the review system problem is not to further complicate it. The simplest solution is usually the best IMHO. But I could be wrong, I've been known to wrong once or twice before.

--
Ja ne, Ja na
Kamenoko

[kamenoko]

@Gambit

The crux of your argument depends on the fact that one cannot tell from reading the first two or three lines whether or not it will be useful. Anyone with the ability to read english at a 6th grade level should be able to determine that.

The thing I am dead set against is allowing other people to decide the value of reviews or even if it's worth my time. If I'm in the miniority with this opinion, then I'll accept a peer review system. Otherwise you moderators need to evaluate the philisophical direction you're taking this database in.

--
Just something to consider,
Kamenoko

[Iceman[grrrr]]

kameneko, keep in mind that everyone could evaluate reviews once they have submitted a number of accepted reviews!

it doesn't really creates 2 category of users since those people don't check every reviews but only a couple chosen at random. And since everything is anonymous, there won't be any biaised votes. If your review is good, it will be accepted anyway!

[Elberet]

Yup, elitism itself is not bad, as long as the elite class is open to everyone who can fulfill certain criteria in a fair test. Since the elite, in our case, does not get any say on who may join them or how others are chosen to become elite as well, the system prevents what you fear: that elitism turns into an oligarchy.

1. The system depends on the previous peer review system, which many have thought is flawed. The continuation of that system is also necessary to make the new system work (how else will the pool of trusted reviewers grow?)

Nope. Once the new system is in place and the CGI knows a few users who it can choose as voters, the old peer ratings can be discarded completely. New possible voters can be chosen simply based on how many reviews they posted. Since all these reviews have to be approved before they are counted, the peer votes are no longer necessary, quality is already assured and the simple fact that 10 reviews were approved is sufficient to trust that a user knows how to write and rate reviews.

Yes, this makes it relatively easy to become a trusted reviewer, one who is considered as a possible voter. But after all, that's the point. I'm not trying to build a superior class, I'm trying to prevent the malicious and really stupid users from pissing off those who carry this community...

2. The system will create two classes, those who can approve reviews, and those who can't. Philisophically I feel this cannot work.

As I said above, it depends on how these classes are built and how you can get into these classes. Even democratic societies have classes, but thanks to democracy and individual freedom, we have vertical mobility: It's possible to get into a different class by one's own decision and work. It's the same with this system. Write good reviews, be a trusted reviewer. The "upper class" can not prevent you from doing so, so vertical mobility and freedom are preserved.

Seeing as how the primary purpose of the anidb is I believe to build and maintain one's anime list,

Funny, I always thought it's purpose was to index and catalogue the anime video files floating through the ed2k network. The mylist is of course a central part, but AniDB has nevertheless become an important resource for anime novices who can count the files they have on a single hand - thus not because of the feauture to index their files, but because of the good reviews and relatively unbiased ratings AniDB has to offer.