Suggestion for improved voting on reviews [DONE]

old granted and denied feature requests

Moderator: AniDB

Keitarou
Posts: 8
Joined: Thu Apr 10, 2003 4:56 pm

Post by Keitarou » Fri Nov 26, 2004 4:55 pm

Phew, logged in for the first time since a long time, because it seems that topic is actually worth some other responses. :)

The "trusted reviewers" idea is imho really bad for something like AniDB. Too elitist, not really consistent with the idea of AniDB, etc. All the reasons were already mentioned, most of them by Don Gato and I second all this opinions. Sure, there are a lot of people voting "no", just because they don't agree with the opinions of the reviewers (I saw that issue for CCS where a few people voted all of the reviews down as well as for other animes), but well, as long as one is pleased with his own review, that shouldn't matter. ;)

On the other hand I really like the system exp-sama suggested.
exp wrote:another idea:

Changing the yes/no vote system to a 1-10 vote system and making it non-anonymous. Meaning the owner of the review will be able to get a list of all users and their votes.
A textfield could be added near the 1-10 vote dropdown which would allow the voter to add some comments to his vote.

Would that help?
Would we need a new profile setting to allow users to make their review votes anonymous again (defaults to off, such entries would then appear in the review vote overview as by user "anonymous")? (Enabling this option might i.e. enforce a non-empty vote comment field)

BYe!
EXP
To remind you of his idea, since the pace of the thread diverged in another direction after that. :)

Ultima
AniDB Staff
Posts: 335
Joined: Tue Oct 01, 2002 11:13 pm
Location: GOTT Head Office, Planet Aineias

Post by Ultima » Fri Nov 26, 2004 5:49 pm

Excuse the big, bold letters because it seems that people don't get where I'm coming from. I'm rather patient by nature, but when others keep thinking that nothing is wrong when there is something blatently wrong I feel hard pressed not to express my feelings.

Sigh, well I'm getting tired of proposing my/requiem's idea, if people deem it too elitist, so be it. I'll just post at Anime-Planet for now on. At least there you won't have stupid voters that vote 'no' on all the reviews simply because it went against their opinion. And please don't say it hasn't happened before because it has happened. For instance, one day most (if not all) Kiminozo reviews suddenly all had a lower rating. WOW, seems like someone voted 'no' on like all the reviews. If you don't believe me you can ask requiem because he's the one that pointed it out to me. "Dude, it seems like some idiot voted 'no' on all the Kiminozo reviews, go take a look". A lo and behold everyone's review ratings were slightly lower.
Keitarou wrote:Sure, there are a lot of people voting "no", just because they don't agree with the opinions of the reviewers (I saw that issue for CCS where a few people voted all of the reviews down as well as for other animes), but well, as long as one is pleased with his own review, that shouldn't matter.
Okay, if you say so. Let's have all reviews end up like Gambit's Berserk review - yeah that's it. Hmm.. a 1.81 rating with 27 votes. WOW, "by gee golly george" review ratings don't matter especially if good ones get send near the bottom - yeah, review ratings don't matter at all. -_-;

No offense Keitarou, I believed that philosophy long ago and it would be okay if good ones remained at the top, but this isn't the case at AniDB. Many AniDB users vote according to their gut instinct and whether it is in accordance with their opinions. Not if the review is helpful or not. You could put these words in big, humongous letters and it wouldn't matter: "You should NOT vote 'NO' because the writer's opinion differs from yours. Instead, vote 'NO' if the review doesn't give info WHY something is good or bad." People at AniDB would still vote the way they feel. It has and always has been like that at AniDB.

I don't care about a reviewer's opinions, I just see if they're helpful or not. And when I see reviews at the bottom of the page when they should be near the top, I think there is something wrong. I dunno... I must be stupid or something, or is something actually wrong with this situation? And furthermore, I've seen some people actually re-write their review because they intitially thought they should go with the "popular opinion" of the crowd. But after some thought, they re-wrote their reviews to reflect their real opinions. I've seen it happen a couple times at AniDB, so I know it exists. How can you have reviewers be afraid to say what they feel when they feel that their review will just be shoved to the bottom? That's rather ridiculous and absurd. However, if you're exempted, and trusted you don't have to worry anymore. You can say, "oh as long as I wrote what I felt and that's all that matters" uh huh, sure.. What happens if your review was rather good and informative, but has a super bad negative rating? Isn't that a little discouraging? Doesn't that hurt your pride? Doesn't that make you feel unappreciated? Won't you even think a little about whether it is even worth it to post your review at AniDB? If you had a shred of pride in your work, then you probably would say YES to all the questions. However, if you answer NO then that means that you think little of your work, and don't care what others think of it. And you probably shouldn't be in this discussion because this is not about keeping the current review system, it is about improving it. Sure the review rating doesn't matter in the big picture. Your review is still there - at the bottom - life is good.

and about exp/nwa's idea.. I like it, but I feel that it'll be short lived. We'll be back to discussing improving reviews all over again.

sothis
Posts: 28
Joined: Tue Jul 22, 2003 4:15 pm

Post by sothis » Fri Nov 26, 2004 6:17 pm

hey guys,

just thought i'd poke my head in and say something here since anime-planet is getting brought up a bit ;)

i first wanted to speak about my own experiences with the review system here, which mirrors what ultima/requiem/some others are saying.

i went off on a random tangent one day and decided to post tons of my reviews here. most of them were for rare series (since i tend to watch lots of those :D) and others were for well known ones such as KGNE, Princess Tutu, etc. most of the reviews i added were the only review for that series.

anyways, though some were definitely not up to par, most of my reviews, i feel, are/were well written, descriptive, and informative. certainly not two sentences long or anything bad. for most of my reviews, i got good ratings. but for quite a few, especially the ones where my opinion might have been different than the norm, i would get terrible ratings for no reason. obviously because people disagreed with the sentiment. after watching this happen over and over, i finally decided to remove all of my reviews and handle it elsewhere.
And again your ego. If people like your reviews they won't care about being rated bad.
here's the thing, though. it's a little discouraging to put a lot of energy into reviews (mine always take at least 15-30 minutes) and then see them shot down because people are immature asshats who want to bring down your rating. keep in mind that if there are a lot of reviews, this also puts yours down on the bottom where people might not even read it. sometimes all it takes is one review for people to know they want to see/not see a series, so guess how much impact mine has at the bottom of 10 reviews? probably zero. is it an ego thing? damn straight it is... if i write a good, thoughtful review, why does mine deserve to be down under a review of someone who wrote 3 lines saying "wow i didnt like this, it sucked" with a -5 rating, just because people are jackasses and want to ruin my score? it's discouraging, period, and it will drive people away.

so i dunno. i understand that you might think wanting to change the system is elitist, but you also should be understanding why people are saying what they are saying. i'm sure for those who are upset, they are wondering why they bother spending time making something they are PROUD of, only to have it be not read and appreciated (potentially, since its at the bottom of the list), marked down, etc. and regardless of if people are talking about anime-planet or not, it *will* make people want to post reviews elsewhere where they are appreciated. if i hadn't made my own system, i still would have deleted them all from anidb and then offered them to somewhere else such as themanime, etc.

now, as far as the my site vs anidb thing that seems to be coming up, i just wanted everyone to know that i didn't make a reviews db to spite anidb (we are partner sites after all) or because i am pissed off at exp or how the anidb system works. anidb has succeeded and DOES succeed for what it is: a very large anime community, run by the community, for the community. this means everyone has a chance at reviewing, voting on series, etc. this is a system many people want, and that's cool. that's why i'm all for trying exp and "the other side"'s way of modifying the review system. for the majority of the people here, the "everyone gets to participate 100%" thing is what's needed, and it works! for my site, i decided to make it sort of half and half... and that works too on my end.

so yeah.. just thought i'd throw that in. i don't think it's wise to just throw away people's opinions as "this is just your ego talking", because obviously quite a few of the lengthy/good reviewers agree with the sentiment, and it isn't always a bad thing that it's related to ego.

User avatar
nwa
AniDB Staff
Posts: 585
Joined: Sat Jun 07, 2003 10:51 am

Post by nwa » Fri Nov 26, 2004 7:03 pm

hehe.. I've thought about writing reviews at anime-planet as well... but saw that you guys use the category system.. (where the review is split into Audio, Visual, Story etc.) and I can't really write reviews this way anymore.

But anyway, there are people who write reviews just for the fun of it, they don't even care about the voting system.. I on the other hand can't stand to see my review getting negative votes, unless it deserves it and a few of my older reviews probably do deserve a 'No'.. but damn.. I spent like 4 hours writing the NTHT review.. maybe my choise of 'words' is different from others (as I do tend to use some vulgar words in my reviews :P )...

As for choosing a system... there's no such thing as a perfect voting system.. but one can have the best possible system... how would that look like? I dunno... perhaps the current system is the best possible.. perhaps the trusted reviewer system, or the 1-10 point system...
Anime-Planet doesn't have many reviewers (atm that is) so they can just put them all up (some animes have 2 reviews, majority have just one) and it's easy for sothis to choose which reviews to put up and which not... however.. anidb does have a lot of review writers... some are good, some arn't... although it wouldn't be hard to filter out the good writers :roll:

pav
Posts: 14
Joined: Sun Jun 06, 2004 12:00 pm

Post by pav » Fri Nov 26, 2004 7:07 pm

Why do you think that this is some assault on democracy, makes AniDB elitist, that reviewers what to be above others and they care only about their ego?

Review rating shouldnt be democratical as much as the general voting for an anime is!

People who write a lot of (good) reviews put a lot time into it, a lot of work and a lot of thought. They deserve respect. Also they are should be rewarded for their work somehow.

AniDB is a source of new animes for people. And reviews usually help them decide wheter to watch a particular anime or not. What they should get in the first place, is the opinion of experienced people who know anime well, who review much and think about what they see.

If a person feels like seeing other reviews/opinions, he/she can easily scroll down to the 'non-professional' reviews section and read them all to their content. He/she can also look at the ratings of the anime, which are the ultimate and democratic indicators. We are not disallowing freedom, everybody can review and write whatever he/she wants.

It is very sad to read a post like that sothis' above mine. But i understand him very well. I once wrote a review of Konomini. It was in the time when it was one of the most popular current animes. But my opinion differed a little from general, so i my review got bashed so much i decided to take it off. And so i have not written any reviews for anidb. Simply because of this. When your work and opinion meets with such a response, you just don't want to say anything more ever again.

Please, take requiem's and Ultima's idea into deep consideration. Once again, it is not disallowing any freedom nor killing 'the whole idea of anidb'.

User avatar
DonGato
Posts: 1296
Joined: Sun Nov 17, 2002 9:08 pm
Location: The Pampas, The land of the Gaucho!
Contact:

Post by DonGato » Fri Nov 26, 2004 7:13 pm

As we will never agree on this we have two options now:

- look for a point in the middle (something like exp already proposed)
- don't care about one of the sides' opinions (being elitists or 'communists')

I don't think anyone's opinion on this will change. This issue was brought many times. You want to make AniDB an elitists lair, just do it! Nobody will stop it. Of course some other people will feel, as you now, discouraged by that and use other sites to seek info.

This is my opinion, not a FACT! So don't blame me for not thinking like you. :roll:

wahaha
AniDB Staff
Posts: 1497
Joined: Sun Nov 17, 2002 3:33 pm

Post by wahaha » Fri Nov 26, 2004 7:14 pm

The "Helpful? Yes/No"-system was added to sort out bad reviews by some kind of peer review. In that respect, it somewhat succeeded since really bad reviews can usually be found near the bottom.

Now, the problem is how to make sure that the good ones aren't there either.

IMO, it wouldn't really help to allow rating reviews from 1-10, at least as far as the problem of "people voting to reflect how much they agree with the review" is concerned. Surprising or not, I'm also in favor of "Trusted Reviewers" to relieve the unneccessary* pressure from those reviewers who proved that they can write proper reviews.
*) They can write proper reviews - that's the sole reason for the "Yes/No"-system to exist, so why apply it to them aswell? I agree with requiem and Ultima here.

Sure, there would still have to be decided who becomes a "trusted reviewer" - and there will probably (again) be much criticism about it; but in general, I'm also in favor of adding it.

User avatar
DonGato
Posts: 1296
Joined: Sun Nov 17, 2002 9:08 pm
Location: The Pampas, The land of the Gaucho!
Contact:

Post by DonGato » Fri Nov 26, 2004 7:17 pm

pav wrote:It is very sad to read a post like that sothis' above mine. But i understand him very well.
No you don't!
Is SHE, not HE! :lol:

EDIT: on the topic, I remember a discussion about having preferred voters. People whom their votes had a greater value than others. So, they can fix some of this deviations without changing the full system (putting 'trusted' people above everyone).

EDIT2: ohhh... and about Gambit's Berserk review, I don't think it deserves such low value but neither to be at the top. I don't care if it was boring for him... anyway, I like the rest of the review.

Signed: Berserk hatter #2.
Last edited by DonGato on Fri Nov 26, 2004 8:08 pm, edited 1 time in total.

pav
Posts: 14
Joined: Sun Jun 06, 2004 12:00 pm

Post by pav » Fri Nov 26, 2004 8:06 pm

DonGato wrote:
pav wrote:It is very sad to read a post like that sothis' above mine. But i understand him very well.
No you don't!
Is SHE, not HE! :lol:
ahhh sothis im terribly sorry
but wtf, you all keep saying on AR im a girl too! :/

sothis
Posts: 28
Joined: Tue Jul 22, 2003 4:15 pm

Post by sothis » Fri Nov 26, 2004 8:53 pm

yay for gender mixups! :D

nwa: just as a note, people do tend to review on anime-planet, but not usually long or thoughtful reviews. the site reviews are linked to a forum thread where anyone can respond (sort of like this two pronged system you guys are discussing here), but there are only a few people besides myself who i allow to go up on the site reviews so far :) then again, not many have tried to submit long/format ones so indeed, there haven't been many to sift through ;)

User avatar
Rar
AniDB Staff
Posts: 1471
Joined: Fri Mar 12, 2004 2:41 pm
Location: UK
Contact:

Post by Rar » Fri Nov 26, 2004 8:55 pm

DonGato wrote:Signed: Berserk hatter #2.
You can get Berserk hats? Cool. Now where's the sothis cathat pic...

Rar

User avatar
requiem
Posts: 42
Joined: Thu Aug 07, 2003 10:27 pm
Location: koko ni inai doko demo ni iru...
Contact:

Post by requiem » Fri Nov 26, 2004 9:18 pm

Summary

As there are PURELY two sides to this argument, I've summed up a list of possible things to agree on.

1) We all agree that the system needs change, right? Though there is an elitist and communist side to this argument, both groups want some change. Therefore, we can:

a. Do nothing. As a result of this, we will continue this argument until the thread is locked and the review system remains unchanged, and quite as unappreciated as before.

b. Be elitist. We develop a trusted reviewers section, test it out, and see if it works. If it works, then the system will gain an exemption pull as well as a party of quality reviewers. If it doesn't, AniDB will become a democratic warground because of a lack of efficiency on the reviewers' part.

c. Be communist. Search and develop a much more stable system that will benefit the community as a whole, and not just the reviewing pool. This would involve, in particular, nwa's/exp's 1-10 voting system, or other suggestions by other users here in at aniDB.

With all of this in mind, we come to point 2.

2) If we apply a new system, what will we need to do? This is important to plan ahead, since this argument is stressing clearly that AniDB will move on (thus, option 1A is most likely obselete by this step.)

Thus:

a. Create a revewing pool. We leave the voting system alone, and add an exempt pool of reviewers. This group will be given the opportunity to post their reviews in the section on an adjacent column, free from the rating weight system (and rather, arranged alphabetically.) To reach this status, a certain number of pre-requisites must be substantiated before being accepted into the group. Perhaps implementing the CREQ system for this attribute to the user might be a handy tool for this job.

b. 1-10 voting system. We follow nwa's/exp's suggestion, and do a rating system similar to the structure of anime voting. The system should prove to be more balanced, including a non-anonymous vote or a forced-text-field-comment by an anonymous voter. Implementing this would be simple, since exp would know what he would want to do with it anyways. :-D

I think the best way to figure out what to do is try something - I'm really up for trying the trusted review option, but exp's voting system probably would be more appropriate for the time being.

Shall we agree on trying exp's system first - say for a month or two - to see if it works?

Ultima
AniDB Staff
Posts: 335
Joined: Tue Oct 01, 2002 11:13 pm
Location: GOTT Head Office, Planet Aineias

Post by Ultima » Fri Nov 26, 2004 9:44 pm

Yeah, to be reasonable I say to try exp's idea first. It woudn't hurt imo.

sothis
Posts: 28
Joined: Tue Jul 22, 2003 4:15 pm

Post by sothis » Fri Nov 26, 2004 9:54 pm


User avatar
DonGato
Posts: 1296
Joined: Sun Nov 17, 2002 9:08 pm
Location: The Pampas, The land of the Gaucho!
Contact:

Post by DonGato » Fri Nov 26, 2004 10:15 pm

Well, most of those who voted NO might not take their time to choose a value and fill an explanation to vote, so I think you will discourage them.
Will see what people write in those commentaries. :P

Locked