Normalized ratings/reviews?

shaydwyrm · Post by **shaydwyrm** » Tue Aug 05, 2003 10:02 pm

I came across this on hotornot.com, which seems to have a pretty reliable rating system (admittedly for a completely unrelated topic) - in the FAQ, they mention that they normalize the ratings:

18. My rating and my votes don't seem to match. What's wrong?

Official HOT or NOT scores are normalized to account for things such as the fact that different people have different voting styles. For instance, some people, for reasons unknown to us, only vote 1-5. Should a 5 from these people count as a 5?
Many standard scoring systems use normalization techniques, including the Olympics and the tests people take to get into college in the US. We still give you the chart, which represents raw data, just for fun.

I don't know how difficult this would be to add to the database, but it seems like a damn good idea to me. Of course, I don't know the mathematics of it, but I could probably find out if there's interest.

Elberet · Post by **Elberet** » Tue Aug 05, 2003 10:22 pm

On an unrelated topic, I think the voting system for the reviews is a little weird. For example: If 9 out of 10 members deem a review useful, then the review must be very well written. The approval system, however, only rates the review an [(9*10 - 1*10)/10] = 8. I think an approval rating based on percentiles would be more useful, in the example above: ("yes" votes) 9 / 10 (total votes) = 0.9 = 90%

DonGato · Post by **DonGato** » Wed Aug 06, 2003 12:11 am

Both suggestions make sense.

Post by **exp** » Wed Aug 06, 2003 4:25 pm

hm,

I don't think there's really a need for the this.
Almost everyone will have at least one anime voted with 9 or 10.
And most ppl will not use the ratings from 1 to 3.
I don't see this should mean that we have to lower the resulting score of all animes.
Yes the range which is actually used is limited, most anime votes will prolly be between 5 and 9, but what would we gain by scaling that?
There are ppl who vote very high and there are ppl who vote rather low but with enough votes added that should not make a difference for the resulting anime rating.
I'd rather wonder about ppl voting for ongoing series.

And about the review rating, that 8 and the 0.9 (9) are actually the same. The only difference is that the first one also allows negative values. So 0 is actually 0.5 (5) in your scheme and -10 is 0.0 (0).
You need to keep in mind that the range is broader than for anime and group votes.
I guess this could be kinda confusing to new ppl though. So i see your point.

BYe!
EXP

Elberet · Post by **Elberet** » Wed Aug 06, 2003 10:13 pm

It's not just that it's confusing, but it also doesn't "look" right. Negative approval ratings seem to express that the review was missing the point and simply contained wrong facts, but that's not necessarily true, after all.

I still lobby for that review feedback thingy tho.

Post by **exp** » Thu Aug 07, 2003 9:25 pm

Elberet wrote:I still lobby for that review feedback thingy tho.

?

BYe!
EXP

Elberet · Post by **Elberet** » Thu Aug 07, 2003 9:30 pm

About this thread...

shaydwyrm · Post by **shaydwyrm** » Thu Aug 07, 2003 9:44 pm

I'd say the messaging system is adequate if you really want to comment on someone's review

Elberet · Post by **Elberet** » Thu Aug 07, 2003 10:02 pm

Yep, that's true. I posted that thread and the second-to-last message before reading the announcement on the main page.

Gambit · Post by **Gambit** » Fri Aug 08, 2003 11:27 am

@EXP I think his main point is that he prefers a rating of 1-10, instead of -10 to 10 ...

-10 sounds really bad, so perhaps you can change it to 1-10 ? Personally I think that sounds a lot more fair as well. -10 is like stabbing someone in the head or so

Perhaps a 'Neutral' option would be okay too.
Yes: Good review
Neutral: Just not good enough to approve / Not bad enough to disapprove.
No: Definitely a lameass review. Should be wiped actually (More NO than yes/neutral would result in a -10, perhaps?)
This way you can seperate the really bad reviews from the 'not so good' reviews.

Elberet · Post by **Elberet** » Fri Aug 08, 2003 12:17 pm

Then why not just rate reviews in the same way as animes, giving them "grades" ranging from 1 ("it's not a review, it's a joke - a lame one") to 10 ("hits the nail on the head AND is fun to read") ? But then I guess, wouldn't that be overkill?

zaufany · Post by **zaufany** » Fri Aug 08, 2003 8:02 pm

The voting review system was created to prevent review spamming. Am I wrong? Two values are enough in my opinion.
http://www.anidb.net/forum/viewtopic.php?t=445

The short voting system isn?t perfect too. However, we have created the description about voting. It should be enough for intelligent people to vote correct.
http://www.anidb.net/forum/viewtopic.php?t=348

shaydwyrm · Post by **shaydwyrm** » Fri Aug 08, 2003 8:48 pm

At the same time, though, an effective system should have plenty of room for the idiocy and laziness of the general population. The point is not to say that 'intelligent people will vote correctly', but rather, 'how can we make the system idiot-proof?'

Elberet · Post by **Elberet** » Fri Aug 08, 2003 8:59 pm

shaydwyrm wrote:The point is not to say that 'intelligent people will vote correctly', but rather, 'how can we make the system idiot-proof?'

Meh! Microsoft tried to do that with Windows, and I'd rather not want to see AniDB go down the same way, being all bubbly, candy and tele tubbies, with assistants, wizards and hint popups everywhere. The voting system for animes is quite ok IMHO, the descriptions besides the votes actually help a lot in finding an adequate vote, and even new users who don't speak English as their mother language should be able to figure out that a "1" is simply not the right vote for an anime they just found a little boring.

But if you insist, how about this: For users who have voted on less then 5 animes, put a confirmation into the voting process. Instead of immediately registering the vote, replace the form with a message like "You are about to vote n. This is an extremely low/high value. Please keep in mind that even a vote of x is still very, very good/bad, and votes below/above that should be reserved for your all-time favorite/most detested animes. Are you sure you wish to cast this vote?" Well, I'm sure someone can come up with a more suiteable reminder text.

(Edit: fixed a typo.)

Elberet · Post by **Elberet** » Fri Aug 08, 2003 9:09 pm

zaufany wrote:The voting review system was created to prevent review spamming. Am I wrong? Two values are enough in my opinion.

I don't think spamming is a problem, since every registered user can only submit one review per anime. In fact, I would like to forward this question to exp: What effects do the votes on reviews have, except the Approved / Unrated / Disapproved grouping ?

But to ponder the problems and various ideas regarding review approving and review voting some more: From a reader's point of view, a review can fall into four categories:

The review was great, fun to read and to the point. The reader approves the review and recommends it.
The review answered most important questions and was to the point. It is approved, but lacks style or some decisive points to be truly recommended.
The review contains some errors or inadequate voting. The reader disapproves of this particular review.
The text presented is not a review, is badly written, impossible to read without falling asleep or describes the anime in a totally wrong way.

This list would indicates that we need four votes: Praise - Approve - Disapprove - Condemn. Furthermore, there would be two ratings: An overall approval rating, made up as the percentage of praises or approvals to the total number of votes, and second a review quality rating, reflecting the ratio of praises to negative votes.