Vote normalization

Forum for discussing AniDB rules & standards. No small talk!

Moderator: AniDB

kamenoko
Posts: 26
Joined: Mon Jan 20, 2003 2:06 pm
Location: Neither here nor there
Contact:

Vote normalization

Post by kamenoko »

If you want truly representative stats in the anidb, some degree of vote normalization is needed. What's going on with genshiken at the moment is absolutely disgusting.

It makes sense from a statistical point of view atleast, though I believe this isn't the first time such a thing was discussed.
CheesyP18
Posts: 33
Joined: Wed Aug 27, 2003 1:44 am

Post by CheesyP18 »

this would help a lot with some of the 10 votes and 1 votes, I believe this will give a more accurate representation of where the anime stands against others
Der Idiot
AniDB Staff
Posts: 1227
Joined: Fri Mar 21, 2003 10:19 am

Post by Der Idiot »

Last edited by Der Idiot on Mon Feb 21, 2005 11:32 pm, edited 1 time in total.
Ultima
AniDB Staff
Posts: 335
Joined: Tue Oct 01, 2002 11:13 pm
Location: GOTT Head Office, Planet Aineias

Post by Ultima »

Well that thread is like almost a year old and didn't really go anywhere. Anyone have good ideas for vote normalization? I think that since the db has grown and that there are many more votes per anime than a year ago so.. I think we should give vote normalization some thought.

edit: Well deridiot found another link: http://www.anidb.net/forum/viewtopic.php?t=2309
Enforcer wrote:imdb.com is using the following algorithm to assure that the titles in the top250 are only those titles which the majority of users considers good, not those where a single user votes a 10.
imdb.com wrote:weighted rank (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C

where:
R = average for the movie (mean) = (Rating)
v = number of votes for the movie = (votes)
m = minimum votes required to be listed in the Top 250 (currently 1250)
C = the mean vote across the whole report (currently 6.8)
Advantages:
- better estimate of top 10-animes
- number of votes gives a rating more weight

Disadvantages:
- some very good animes drop out of the list because only few people voted for it
- more stress for the db-server

I think this would be more interesting than a top10 where "only" the average of votes is taken into account.

So long,
Enforcer
imokie wrote:IMDb also only considers votes from regular voters. Perhaps that would be a good idea for AniDB too. It would eliminate the votes from some of the 'fanboys' that only vote once or twice to give the favourite anime a 10.
CheesyP18
Posts: 33
Joined: Wed Aug 27, 2003 1:44 am

Post by CheesyP18 »

exp never looked into it, or maybe he did but didnt give an opinion...
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

I guess I will post my opinion [again]...
  • Votes should from "legitimate users" should not be normalized, given enough user that vote, things will work themselves out.
  • Votes from dummy users created just to vote should be filtered out.
  • IF votes are filtered and/or normalized then on the anime page, the raw votes average (the one without the filtering/normalization applied) should be listed as well.
  • IF a user has their votes filtered there should be a clear indication on the myplace and myvotes screens with [a link to] the list of requirements to have their votes counted in the overall statistics.
The next question is, how do we define "legitimate users". This is a sticky thing to do, but rather maybe we should define some criteria that need to be met before a user's votes can be counted for overall statistics. Here is what I propose (the first two are based on the hint):
  1. User must have X anime in mylist
  2. User must have X [permanent?] votes for anime
  3. User must have no more than 10% '10' votes
  4. User must have no more than 10% '1' votes
Rules 1 & 2 would filter out all but the most persistent users who create multiple accounts just have the votes... I don't really agree with the last two, but similar things have been proposed before and this is simpler and more equitable. Users like this would pass the first two rules.
Elias
Posts: 242
Joined: Tue Feb 17, 2004 4:55 pm

Post by Elias »

Rule number 1. - must be postponed - until AOM 0.6 (or precisely generics for all titles, AOM can be delayed forever) appear.
I would add a rule with minimal time from account activation (a week maybe).
And i would definitely not show all the rules (only give some general description, why some users are not counted - not specifying 10% of 10, 10% of 1, ...
- knowing exact rules will just made it little longer, to boost some title, and sink another:
not only vote 1 and 10, but add them both to mylist and made 9 fake 5 votes for some other titles).
Rar
AniDB Staff
Posts: 1471
Joined: Fri Mar 12, 2004 2:41 pm
Location: UK
Contact:

Post by Rar »

Until we have generics for all titles eh? Looked at anidb recently? :D

Rar
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

Me and my big $#*&@ mouth. I really didn't want to be dragged into this conversation...

As my father likes to say, "Locks are only meant to keep honest people honest." No matter what scheme we come up with, people will be able to get around it if they really want to. We are just trying to make it so that there are some standards before using the votes.

[sarcasm]I know, we should not allow votes from anyone who does not fit a perfect bell curve. Or maybe we only accept votes if they have exactly 10% of each vote.[/sarcasm]

I don't think a time interval matters, after a week, the votes will automatically start counting at that time without the user doing anything. Who cares if it counts today or next week, ultimately it will still count if all the other criteria are met.

The reasons for posting the rules is so that someone who is trying to do the right thing can tell what they are doing wrong if they happen to be filtered out. People would be able to find the discussions in the forum anyway so they would at least have an idea of what exp implements.
Elias
Posts: 242
Joined: Tue Feb 17, 2004 4:55 pm

Post by Elias »

Rar wrote:Until we have generics for all titles eh? Looked at anidb recently? :D
Allmost every day, but after i've got generics i needed, i haven't looked at that part (they are hidden by default). Why wasn't such success annouced?
kamenoko
Posts: 26
Joined: Mon Jan 20, 2003 2:06 pm
Location: Neither here nor there
Contact:

Post by kamenoko »

There are 78 '1' votes for RuroKen reminiscence, if I ran this place, I'd delete those people's accounts, go around to each of their homes around the globe, and piss on their computers until I saw sparks coming out of the case.

But that's just me, I'm pretty sure vote normalization would be a much better plan :D.

I defy any of you to prove to my satisfaction that the RuroKen OVA deserves a 1 :)
donfischi
Posts: 6
Joined: Tue Dec 30, 2003 11:58 pm

Re: Vote normalization

Post by donfischi »

fist of all:
kamenoko wrote:What's going on with genshiken at the moment is absolutely disgusting.
a anime that you don´t like(that much) makes it in the top ten and you call that digusting? i see other things in the top ten that are closer to disgusting then that. and that is downvoting other shows to make one take the lead. that might be nearer to disgusting if the word is at all appropiate in this context.

i for my share would say it is better than other shows in the top ten imo. is it in my personal top ten? it might be close to it but before i can say that i have to wait a little longer and judge what time does about my judgement of this show.

but that aside i also think that the top ten are baised at some points. i think that the worst things in the votes are the votes that are 1 or 2 since punish a great other show and not only promote the show the "cultrip" likes.
they are also the only thing that can be taken care of in a reasonable way, one can sort out extremes mathematically. but that is only possible with the low votes in anidb since there are in most cases (and that are the important ones) no high extremes.
the high votes are to some extend ok even if they might be fanboy influenced, because you cannot forbid someone to like a show or assume he is not truthfull about his high vote. ( on the contrary i think they are absolutly honest even if not based on knowledge or whatever else makes a vote "good") what is nearly as bad as downvoting are the fake accounts. they are maybe the second thing that can be taken care of with reasonable matters, like making some min requirements before a vote is taken into consideration.

what i do not like at all is the weighted vote.
Enforcer wrote:Advantages:
- better estimate of top 10-animes
- number of votes gives a rating more weight
i don´t see the advantage in the first point, or to be precise i don´t see why they are more "estimate" if i take that right for more "credible" or "better".

to the second poin i have to say, that i would not really like a vote to have more wight because of the number of votes, because that would benefit some major poplular fanboy shows that are imo overrated. and some real gems that are really good but that just too few people know about have a really hard time to get into the top ten. there has to be a minimum number of votes to get into the top ten but that is implemented already.

so to make it short: if there should be changes, imo the only ones that would make sense is to sort too low votes out and to take fake accounts not into consideration.
kamenoko
Posts: 26
Joined: Mon Jan 20, 2003 2:06 pm
Location: Neither here nor there
Contact:

Post by kamenoko »

donfischi wrote:a anime that you don´t like(that much) makes it in the top ten and you call that digusting? i see other things in the top ten that are closer to disgusting then that. and that is downvoting other shows to make one take the lead. that might be nearer to disgusting if the word is at all appropiate in this context.
ano... I was talking about the the people who vote 1 for animes for no other reason than to try and knock an anime out of the top 10. I suggest they are FMA fans :D

Do I think Genshiken is one of the top 10 animes of all time? Good god no, but it's there at the moment, and doesn't deserve 1's just because some fanboys anime got knocked out of the top 10. That's kind of why I started this thread in the first place ;)
DonGato
Posts: 1296
Joined: Sun Nov 17, 2002 9:08 pm
Location: The Pampas, The land of the Gaucho!
Contact:

Post by DonGato »

As I said before I agree that some normalization should be done. One time voters or the ones overusing 1 and 10 should be removed from the calculations. High deviations might be interresting to study and handle. Like 500 votes over 8 and 5 bellow 2.
OnegaiNL
Posts: 80
Joined: Wed Oct 20, 2004 4:38 pm

Post by OnegaiNL »

egg wrote:I guess I will post my opinion [again]...
  • Votes should from "legitimate users" should not be normalized, given enough user that vote, things will work themselves out.
  • Votes from dummy users created just to vote should be filtered out.
  • IF votes are filtered and/or normalized then on the anime page, the raw votes average (the one without the filtering/normalization applied) should be listed as well.
  • IF a user has their votes filtered there should be a clear indication on the myplace and myvotes screens with [a link to] the list of requirements to have their votes counted in the overall statistics.
The next question is, how do we define "legitimate users". This is a sticky thing to do, but rather maybe we should define some criteria that need to be met before a user's votes can be counted for overall statistics. Here is what I propose (the first two are based on the hint):
  1. User must have X anime in mylist
  2. User must have X [permanent?] votes for anime
  3. User must have no more than 10% '10' votes
  4. User must have no more than 10% '1' votes
Rules 1 & 2 would filter out all but the most persistent users who create multiple accounts just have the votes... I don't really agree with the last two, but similar things have been proposed before and this is simpler and more equitable. Users like this would pass the first two rules.
NANI, YOUR USING ME AS AN EXAMPLE XD
and btw i would pass all 4 rules, changed alot lol

but yea i agree with your stuff, oh yeah
User must have the anime complete might help, because most ghost accounts don't even add files lol.
Locked