Vote normalization
Moderator: AniDB
Vote normalization
If you want truly representative stats in the anidb, some degree of vote normalization is needed. What's going on with genshiken at the moment is absolutely disgusting.
It makes sense from a statistical point of view atleast, though I believe this isn't the first time such a thing was discussed.
It makes sense from a statistical point of view atleast, though I believe this isn't the first time such a thing was discussed.
Last edited by Der Idiot on Mon Feb 21, 2005 11:32 pm, edited 1 time in total.
-
- AniDB Staff
- Posts: 335
- Joined: Tue Oct 01, 2002 11:13 pm
- Location: GOTT Head Office, Planet Aineias
Well that thread is like almost a year old and didn't really go anywhere. Anyone have good ideas for vote normalization? I think that since the db has grown and that there are many more votes per anime than a year ago so.. I think we should give vote normalization some thought.
edit: Well deridiot found another link: http://www.anidb.net/forum/viewtopic.php?t=2309
edit: Well deridiot found another link: http://www.anidb.net/forum/viewtopic.php?t=2309
Enforcer wrote:imdb.com is using the following algorithm to assure that the titles in the top250 are only those titles which the majority of users considers good, not those where a single user votes a 10.
Advantages:imdb.com wrote:weighted rank (WR) = (v ÷ (v+m)) × R + (m ÷ (v+m)) × C
where:
R = average for the movie (mean) = (Rating)
v = number of votes for the movie = (votes)
m = minimum votes required to be listed in the Top 250 (currently 1250)
C = the mean vote across the whole report (currently 6.
- better estimate of top 10-animes
- number of votes gives a rating more weight
Disadvantages:
- some very good animes drop out of the list because only few people voted for it
- more stress for the db-server
I think this would be more interesting than a top10 where "only" the average of votes is taken into account.
So long,
Enforcer
imokie wrote:IMDb also only considers votes from regular voters. Perhaps that would be a good idea for AniDB too. It would eliminate the votes from some of the 'fanboys' that only vote once or twice to give the favourite anime a 10.
I guess I will post my opinion [again]...
- Votes should from "legitimate users" should not be normalized, given enough user that vote, things will work themselves out.
- Votes from dummy users created just to vote should be filtered out.
- IF votes are filtered and/or normalized then on the anime page, the raw votes average (the one without the filtering/normalization applied) should be listed as well.
- IF a user has their votes filtered there should be a clear indication on the myplace and myvotes screens with [a link to] the list of requirements to have their votes counted in the overall statistics.
- User must have X anime in mylist
- User must have X [permanent?] votes for anime
- User must have no more than 10% '10' votes
- User must have no more than 10% '1' votes
Rule number 1. - must be postponed - until AOM 0.6 (or precisely generics for all titles, AOM can be delayed forever) appear.
I would add a rule with minimal time from account activation (a week maybe).
And i would definitely not show all the rules (only give some general description, why some users are not counted - not specifying 10% of 10, 10% of 1, ...
- knowing exact rules will just made it little longer, to boost some title, and sink another:
not only vote 1 and 10, but add them both to mylist and made 9 fake 5 votes for some other titles).
I would add a rule with minimal time from account activation (a week maybe).
And i would definitely not show all the rules (only give some general description, why some users are not counted - not specifying 10% of 10, 10% of 1, ...
- knowing exact rules will just made it little longer, to boost some title, and sink another:
not only vote 1 and 10, but add them both to mylist and made 9 fake 5 votes for some other titles).
Me and my big $#*&@ mouth. I really didn't want to be dragged into this conversation...
As my father likes to say, "Locks are only meant to keep honest people honest." No matter what scheme we come up with, people will be able to get around it if they really want to. We are just trying to make it so that there are some standards before using the votes.
[sarcasm]I know, we should not allow votes from anyone who does not fit a perfect bell curve. Or maybe we only accept votes if they have exactly 10% of each vote.[/sarcasm]
I don't think a time interval matters, after a week, the votes will automatically start counting at that time without the user doing anything. Who cares if it counts today or next week, ultimately it will still count if all the other criteria are met.
The reasons for posting the rules is so that someone who is trying to do the right thing can tell what they are doing wrong if they happen to be filtered out. People would be able to find the discussions in the forum anyway so they would at least have an idea of what exp implements.
As my father likes to say, "Locks are only meant to keep honest people honest." No matter what scheme we come up with, people will be able to get around it if they really want to. We are just trying to make it so that there are some standards before using the votes.
[sarcasm]I know, we should not allow votes from anyone who does not fit a perfect bell curve. Or maybe we only accept votes if they have exactly 10% of each vote.[/sarcasm]
I don't think a time interval matters, after a week, the votes will automatically start counting at that time without the user doing anything. Who cares if it counts today or next week, ultimately it will still count if all the other criteria are met.
The reasons for posting the rules is so that someone who is trying to do the right thing can tell what they are doing wrong if they happen to be filtered out. People would be able to find the discussions in the forum anyway so they would at least have an idea of what exp implements.
There are 78 '1' votes for RuroKen reminiscence, if I ran this place, I'd delete those people's accounts, go around to each of their homes around the globe, and piss on their computers until I saw sparks coming out of the case.
But that's just me, I'm pretty sure vote normalization would be a much better plan .
I defy any of you to prove to my satisfaction that the RuroKen OVA deserves a 1
But that's just me, I'm pretty sure vote normalization would be a much better plan .
I defy any of you to prove to my satisfaction that the RuroKen OVA deserves a 1
Re: Vote normalization
fist of all:
i for my share would say it is better than other shows in the top ten imo. is it in my personal top ten? it might be close to it but before i can say that i have to wait a little longer and judge what time does about my judgement of this show.
but that aside i also think that the top ten are baised at some points. i think that the worst things in the votes are the votes that are 1 or 2 since punish a great other show and not only promote the show the "cultrip" likes.
they are also the only thing that can be taken care of in a reasonable way, one can sort out extremes mathematically. but that is only possible with the low votes in anidb since there are in most cases (and that are the important ones) no high extremes.
the high votes are to some extend ok even if they might be fanboy influenced, because you cannot forbid someone to like a show or assume he is not truthfull about his high vote. ( on the contrary i think they are absolutly honest even if not based on knowledge or whatever else makes a vote "good") what is nearly as bad as downvoting are the fake accounts. they are maybe the second thing that can be taken care of with reasonable matters, like making some min requirements before a vote is taken into consideration.
what i do not like at all is the weighted vote.
to the second poin i have to say, that i would not really like a vote to have more wight because of the number of votes, because that would benefit some major poplular fanboy shows that are imo overrated. and some real gems that are really good but that just too few people know about have a really hard time to get into the top ten. there has to be a minimum number of votes to get into the top ten but that is implemented already.
so to make it short: if there should be changes, imo the only ones that would make sense is to sort too low votes out and to take fake accounts not into consideration.
a anime that you don´t like(that much) makes it in the top ten and you call that digusting? i see other things in the top ten that are closer to disgusting then that. and that is downvoting other shows to make one take the lead. that might be nearer to disgusting if the word is at all appropiate in this context.kamenoko wrote:What's going on with genshiken at the moment is absolutely disgusting.
i for my share would say it is better than other shows in the top ten imo. is it in my personal top ten? it might be close to it but before i can say that i have to wait a little longer and judge what time does about my judgement of this show.
but that aside i also think that the top ten are baised at some points. i think that the worst things in the votes are the votes that are 1 or 2 since punish a great other show and not only promote the show the "cultrip" likes.
they are also the only thing that can be taken care of in a reasonable way, one can sort out extremes mathematically. but that is only possible with the low votes in anidb since there are in most cases (and that are the important ones) no high extremes.
the high votes are to some extend ok even if they might be fanboy influenced, because you cannot forbid someone to like a show or assume he is not truthfull about his high vote. ( on the contrary i think they are absolutly honest even if not based on knowledge or whatever else makes a vote "good") what is nearly as bad as downvoting are the fake accounts. they are maybe the second thing that can be taken care of with reasonable matters, like making some min requirements before a vote is taken into consideration.
what i do not like at all is the weighted vote.
i don´t see the advantage in the first point, or to be precise i don´t see why they are more "estimate" if i take that right for more "credible" or "better".Enforcer wrote:Advantages:
- better estimate of top 10-animes
- number of votes gives a rating more weight
to the second poin i have to say, that i would not really like a vote to have more wight because of the number of votes, because that would benefit some major poplular fanboy shows that are imo overrated. and some real gems that are really good but that just too few people know about have a really hard time to get into the top ten. there has to be a minimum number of votes to get into the top ten but that is implemented already.
so to make it short: if there should be changes, imo the only ones that would make sense is to sort too low votes out and to take fake accounts not into consideration.
ano... I was talking about the the people who vote 1 for animes for no other reason than to try and knock an anime out of the top 10. I suggest they are FMA fansdonfischi wrote:a anime that you don´t like(that much) makes it in the top ten and you call that digusting? i see other things in the top ten that are closer to disgusting then that. and that is downvoting other shows to make one take the lead. that might be nearer to disgusting if the word is at all appropiate in this context.
Do I think Genshiken is one of the top 10 animes of all time? Good god no, but it's there at the moment, and doesn't deserve 1's just because some fanboys anime got knocked out of the top 10. That's kind of why I started this thread in the first place
NANI, YOUR USING ME AS AN EXAMPLE XDegg wrote:I guess I will post my opinion [again]...
The next question is, how do we define "legitimate users". This is a sticky thing to do, but rather maybe we should define some criteria that need to be met before a user's votes can be counted for overall statistics. Here is what I propose (the first two are based on the hint):
- Votes should from "legitimate users" should not be normalized, given enough user that vote, things will work themselves out.
- Votes from dummy users created just to vote should be filtered out.
- IF votes are filtered and/or normalized then on the anime page, the raw votes average (the one without the filtering/normalization applied) should be listed as well.
- IF a user has their votes filtered there should be a clear indication on the myplace and myvotes screens with [a link to] the list of requirements to have their votes counted in the overall statistics.
Rules 1 & 2 would filter out all but the most persistent users who create multiple accounts just have the votes... I don't really agree with the last two, but similar things have been proposed before and this is simpler and more equitable. Users like this would pass the first two rules.
- User must have X anime in mylist
- User must have X [permanent?] votes for anime
- User must have no more than 10% '10' votes
- User must have no more than 10% '1' votes
and btw i would pass all 4 rules, changed alot lol
but yea i agree with your stuff, oh yeah
User must have the anime complete might help, because most ghost accounts don't even add files lol.