Suggestion for improved voting on reviews [DONE]

Lion · Post by **Lion** » Fri Dec 12, 2003 3:32 pm

EDIT from wahaha:
This post is a reply to the suggestions made in "Review Moderation & Trusted Users/Reviewers". As Lion's suggestions seem to be a accepted as a compromise, I decided to give it a fresh start.

As a short explanation for new readers:
You'll find the abbreviation "TR" being used in the following posts. It refers to "Trusted Reviewers", a group/classification that doesn't exists yet, but was suggested to be formed from the current (and future) reviewers in AniDB in one way or the other, as some kind of "expert-group".

Well, what I can say, I read all the posts and surely I missed something

, but I wanted to try adding my two cents to the discussion, hopefully without adding more problems.

I like the idea of changing the review system, I really hate animenfo's reviews, and when I see something like it on anidb it piss me off a bit.

And again I hate to see that a review has fallen down only because it is against an anime, perhaps a famous one that fanboys really like (this hasn't happened to me, or I think so

).

On the other side, with the going on of the discussion I felt that these TR where becoming more and more like censorship, and if it is true that this could prevent bad reviews to show on top of pages, or even to show, it doesn't suit my taste of doing things.

I thought a bit about all the discussion, and I came up with this idea:
the review system remains pretty much the same, but with a slight difference; there are the TR (elected, sorted by review rating or number or both, your choose) that CAN'T vote for the review. They receive a notice for voting about a review, let's say ten of the TR, and only after these ten people (choosen randomly) have expressed their vote, automatically to the review is added (or subtracted) x points (where x is to be studied how much big it should be). This way, even if a review is still visible, it would be high (or down) the reviews' list, being influenced by the stronger vote of a full group of TRs (people that we suppose are able to judge a review).

I think it should be good that reviews that are lower than a certain score should be shown in another page, let's say we have a link on the bottom of the reviews' list that says "show other reviews".

Plus, I think that a review that gets a really bad score should be deleted after a while. Given the fact that the passage between a "second page review" and a "to be deleted" review shouldn't be so fast (or if it is, then surely that review it's just trash), the reviewer has all the time he need to edit his review and repost it. Of course TR vote shouldn't be so big that could put a review from first page directly in the "to be deleted" list, so only TR vote together with the votes of some "normal" users could sign the death of a review.

I think this way TR judgement is important (but not imperative, 'cause a big mass of people could still influence the review in a way or another) and bad reviews would be put away after a bit of time, not disturbing the reading of the good ones.

As I said, just my two cents.

DonGato · Post by **DonGato** » Fri Dec 12, 2003 5:43 pm

I think yours is an excellent idea. The value should not be too much as you say but the rest seems to be a good solution.

Gambit · Post by **Gambit** » Sat Dec 13, 2003 2:47 pm

I`ll join DonGato, this sounds like a good idea

Lion wrote:I think it should be good that reviews that are lower than a certain score should be shown in another page, let's say we have a link on the bottom of the reviews' list that says "show other reviews".

Let`s say that a rating of below 0 after 5 votes is bound to get to the other side? Or perhaps it should be after 10 votes, considering the number of TRs.

Lion wrote:Plus, I think that a review that gets a really bad score should be deleted after a while. Given the fact that the passage between a "second page review" and a "to be deleted" review shouldn't be so fast (or if it is, then surely that review it's just trash), the reviewer has all the time he need to edit his review and repost it. Of course TR vote shouldn't be so big that could put a review from first page directly in the "to be deleted" list, so only TR vote together with the votes of some "normal" users could sign the death of a review.

So, let`s say that the 'vote' of a TR counts like, 2-4 votes ? Or do you mean that the average of all TRs who get the notice counts as 1 vote? In that case it`ll have to be quite significant stronger.

I`d suggest to delete the bad review after it receives more than 20 no-votes and is way below the 0-rating, like -5.

DonGato · Post by **DonGato** » Sat Dec 13, 2003 3:17 pm

I think a vote of a TR can have a better value. Let's say 2x or 3x the rating of a normal user in relation to the number of TRs, if more than 5 I think we shouldn't go above 2x. Remember most reviews have less than 10 user votes for now... and some of them came from to be TRs.

About bad reviews, I would keep them unless the have an extremely low rating (I believe I've seen some -10 that means nobody though it was useful). Those should be removed after... a month? (doj!... just read again Gambit proposal and seems to be ok as well with the addition of a number of votes).

Bad reviews can be shown in a different page linked from the main page (that was a request on the the first review system proposal), but new ones and good ones should be shown in the main page. What I don't know which ones come first.

Gambit · Post by **Gambit** » Sat Dec 13, 2003 3:37 pm

DonGato wrote:About bad reviews, I would keep them unless the have an extremely low rating (I believe I've seen some -10 that means nobody though it was useful). Those should be removed after... a month? (doj!... just read again Gambit proposal and seems to be ok as well with the addition of a number of votes).

I don`t even mind keeping those reviews, as long as they`re not shown on the main review-page. But it is indeed useless to keep those -10 reviews around after 10 people voted the same way for that review. It`ll only clutter the database after all.

DonGato wrote:Bad reviews can be shown in a different page linked from the main page (that was a request on the the first review system proposal), but new ones and good ones should be shown in the main page. What I don't know which ones come first.

Newest one on top, the rest sorted by rating, would be the best way to display them, imho.

I wonder if Elberet`s part will be used here as well. I mean the part where he said something like this:

A TR doesn`t give out a vote, he gets the options 'Excellent', 'above average', 'decent enough', 'below average' and 'Completely useless', which compares with 9-7-6-5-3 (or something alike). He then picks the one which suits the review best and then the result of this combination for all TR`s will be used as the 'TR-rating'.

I think a rating according to text is better than according to marks. In time you can even think of changing the normal users to this review-rating instead of the 1-10 ratings.

[edit]
Okay, in fact that was my idea which resulted in this after reading Elberet`s post for the voting-system with rating. I do think this textual voting is better than just disapproving/approving since some people think of a 7 as good, while others distinguish between 'good' (8) and 'above average' (7).
[/edit]

DonGato · Post by **DonGato** » Sat Dec 13, 2003 4:49 pm

That's why we have text counterparts in the anime voting system as well. Numbers in ratings doesn't mean the same for all people so or we add a text at the side or we use textual ratings instead.

Gambit · Post by **Gambit** » Sat Dec 13, 2003 5:25 pm

I know, but I guess I`m saying it the wrong way what I mean :/
Ah well, never mind.

DonGato · Post by **DonGato** » Sat Dec 13, 2003 7:00 pm

Just remembered you can vote Yes/No only there. There is a plan to change that also?

Lion · Post by **Lion** » Sun Dec 14, 2003 9:59 am

Here I am again

I thought this system about the voting of the TRs:
When the TRs are called to a vote, they can express 4 different judgements:
+2- Very good review
+1- good review
-1- bad review
-2- only crap.

Let's say that the TRs requested to vote are 5. They submit their vote, and once all of them have voted, the final result is a number between -10 and +10. Depending on the sign of the resulting number to the review in question is applied that number of votes as yes/no votes.
For example if the final judgement of the TRs is, let's say 7, then to the review are added 7 "yes votes".

All this should give reviews a good boost up (or down). In fact, thinking about that reviews are unlikely to get more than ten votes (and I think a good bunch of them are from the future TRs, that in my mind shouldn't be able to express a normal "yes/no" vote), the weight of the TRs decision will surely influence the review rating. Plus, this way if the TRs themselves can not decide if a review is good or bad, resulting in a number near 0, this will bring very little votes to the review, lefting it to the judgement of normal users.

Last, if a review is really good (or is really trash), the final TRs' vote it's likely to be +10 (-10), so this will add 10 "yes" ("no") votes to the review. I think that even if there are a bunch of funboys that really like an anime, this value should be sufficient to weight well a review for a value close to what it deserves.

I think that reviews that get a negative rating (in total, not only by the TRs) should be put in a second page.
And if a review has got at last 15 votes (this way even if all TRs voted -2 for it, the total number of votes would be 10) and has at last -5 as resulting rating, it should be deleted. The scenario is this:
all TRs vote "-2", so the review gets a -10 as rating. If 5 more people vote yes, the review rating goes up to -3,33 and so it will not be deleted, even if it is still on the second page (I think this is unlikey to happen, 'cause if all TRs voted -2 you have to find a REAL DIEHARD fanboy to vote yes for that one, and even if you find one, there are the other 4s).
Notice that is enough to have 4 of the other 5 people to vote yes to save a review from delete (result should be -4,6), even if of course that review is really near to his death (but probably, again, it deserves it, getting all this bad votes).

Last (I promise it

): TRs shouldn't be able to see other TRs' votes, as to not be influenced by them.

wahaha · Post by **wahaha** » Sun Dec 14, 2003 12:03 pm

Lion wrote:And if a review has got at last 15 votes (this way even if all TRs voted -2 for it, the total number of votes would be 10) and has at last -5 as resulting rating, it should be deleted. [...]

I think - for now - the discussion on when a review could be deleted should be postponed. Even for bad reviews, this is the most extreme measure to deal with them.
Also to have a better understanding of how the rating-based deletion would work in practice - if anything - I'd rather have these reviews on a third page or marked in a special way, until it'll become evidently neccessary that get rid of them.

Post by **nwa** » Mon Dec 15, 2003 9:55 am

well...what can I say, such a simple solution and none of us couldn't see it until Lion came and put everything into place

Lion wrote:TRs shouldn't be able to see other TRs' votes, as to not be influenced by them.

well, I don't know a voting system that would show the vote of a person so that is only natural, although there will most probably be some talk about the votes in IRC

Gambit · Post by **Gambit** » Mon Dec 15, 2003 11:16 pm

nwa wrote:well, I don't know a voting system that would show the vote of a person so that is only natural, although there will most probably be some talk about the votes in IRC

... or even fix them ... *whistles innocently when walking past nwa*

Nah, but it`s a good idea, I`m just too tired to make a good remark now, but I`m a supporter of the idea.

I`m just against the 4 ways of a vote. What if it`s neither a good nor a bad review? It has some good points, but it`s also not good enough to be voted good - but neither is it bad enough to be called 'bad'. You`ll need a midway for that, and I believe I already gave a suggestion for that one.

DonGato · Post by **DonGato** » Tue Dec 16, 2003 12:03 am

Gambit wrote:I`m just against the 4 ways of a vote. What if it`s neither a good nor a bad review? It has some good points, but it`s also not good enough to be voted good - but neither is it bad enough to be called 'bad'. You`ll need a midway for that, and I believe I already gave a suggestion for that one.

Do not vote?

requiem · Post by **requiem** » Wed Nov 24, 2004 3:41 am

Hey minna,

After several months of letting this issue rest, I think it's time we raise it again - since now I'm getting honestly more and more irritated with the reviewing system.

As I complained before, people are still voting for reviews based on what they do or don't like. I didn't mind this since people weren't singling me out...but now:

http://anidb.ath.cx/perl-bin/animedb.pl ... s&uid=1816

If it changed, today my avg. rating is 7.06 after 24 hours - 16 hours ago it was 6.97. Yesterday it was 8.19. This is simply because of a few reviews with horrible negative scores. I managed to ask a few people to read my review and vote to see if I could recover...and I really don't give a damn as long as I don't have any red. I really suspect that it is all the same person voting "no" for anything I write...

Thus, I'm angry. If this person decides to negate anything I write, I'm supposed to live with this? So...

Maybe my reviews suck. After all, I do extended research and even update and change things according to criticism...

And maybe they actually really do suck - but they coherent, non-spoiling, carefully planned out reviews that I want other people to like; just cause one doesn't shouldn't mean my overall rating as a reviewer should be bad.

Yes, it bothers me. I don't know if Gambit, vivafruits, pantha, Ultima, or many of the other reviewers would feel this way or not, but when you put that much work into a website and someone shits all over you like that you feel cheated. I can get over it over time, I'm sure...but this is really painful no matter how much I minimize it. I feel offended since the offender didn't even have the dignity to PM me that I'm a 'pathetic reviewer" in the least.

Maybe I'm overreacting. I have enough written here to pass out of K-12 English curriculum at the school system here in the US. Gambit already has written enough to publish an Anime Reviewing Bible. Though he hasn't said anything, I think the fact that both of us have been leveled by ignorant voters is a justified reason to overreact.
I want some privileges. I find it funny that moderators rule the populace at Animereactor, but according to the last topic of discussion we had on this subject, it is bad to have a ruling "body of reviewers" permitted here at AniDB. I'm usually really patient and humble when it comes to expressing my feelings on an issue, but nothing has changed about this system for the better. I'm not asking for mod rights or anything, but I would like a little bit of separation from the mob who on some occasions don't even know what anime they are reviewing, they just know it's cool!

I am not panicked, upset, or desiring vengeance - I just am disappointed. Sites like Anime-Planet may not fit exactly to my style, but will at least offer me a better system of reviewing in the long run; am I just better off there, or is there anything we can do?

Ultima · Post by **Ultima** » Wed Nov 24, 2004 5:13 am

Well requiem, I do share your sentiments about AniDB's review system. Reviews have become much better and well-written ever since the new review voting system was implemented, but it still needs some sort of revision. I don't write many reviews at AniDB simply because users vote on personal opinion and don't care whether the review was helpful or not. Like requiem, I do put forth a great deal of thought and effort to review anime and sometimes they end up at the bottom of the reviews simply because it went against the accepted opinion. Like my Saikano review, Gambit's Berserk review, and requiem.. well I know there are a few reviews that he's written that deserve better.

I think there should be a system or something where once a user has written a certain # of reviews, that they be candidate for reviewing anime without having to be rated. If a reviewer has shown enough time and dedication to writing reviews, there's no reason why that reviewer has to be subjected to the sometimes cruel votings. For example, say once a reviewer has written 25 reviews (imaginary number) AniDB will automatically nominate them to a special reviewer's pool at AniDB. Moderators can look back on that user's past reviews and decide whether they deserve special privileges. If they are accepted, then all their reviews will take precedence over all other reviews at AniDB because they are recognized by the AniDB staff as a good reviewer. Furthermore, that user's reviews will no longer be subjected to the review voting system and remain at the top of the list.

Dedicated reviewers should be rewarded for their efforts. It's not like they get paid money or anything, they do it out of their love of anime and writing. It certainly would be in AniDB's best interest to reward dedicated reviewers because those type of reviewers reflect AniDB's image. If those reviewers stop posting at AniDB because of faulty reviewer ratings, that would be most unfortunate.