New rating calculations/system

All your questions about AniDB belong in here.
No download support!

Moderator: AniDB

User avatar
ohtoriakio
Posts: 11
Joined: Fri Apr 28, 2006 10:45 pm

Post by ohtoriakio » Sat Jan 13, 2007 8:48 am

Dunno if this is due to the new rating system or something else altogether but a couple of votes on my reviews seem to have been "ghosted". They are there when I look at the feedback I received but they aren't on my main review page. This also caused one of my reviews that reached the 5 votes mark to not make it out of the "Unrated" section, is there something I can do about that? Thanks for your time.

Der Idiot
AniDB Staff
Posts: 1227
Joined: Fri Mar 21, 2003 10:19 am

Post by Der Idiot » Sat Jan 13, 2007 10:07 am

laxxy wrote:
pelican wrote:Yes, we could tell you. No, we're not going to. (And KAA happen to be pretty good among dvd rippers, which admittedly isn't saying much. Recent KAA anyway.)
Speaking about groups, I think there is definitely a problem: you've got the first place occupied by WF/KAA, who made all of 4 episodes of one series together before quitting. There needs to be some adjustment for the # of votes, or # of episodes subbed, or smth like that.

that was already fixed in the next version which is not live yet

laxxy
Posts: 20
Joined: Thu Mar 02, 2006 4:34 pm

Post by laxxy » Sat Jan 13, 2007 1:30 pm

one other thing -- you quite understandably do not want the ratings live, but i think it would be nice to update them at least once a day or two.

User avatar
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Post by exp » Sat Jan 13, 2007 4:00 pm

laxxy wrote:one other thing -- you quite understandably do not want the ratings live, but i think it would be nice to update them at least once a day or two.
that's actually a bug. we're having some problems with cache invalidation atm. so you're quite often seeing stale, cached ratings.

BYe!
EXP

vivafruit
Posts: 17
Joined: Mon Jan 19, 2004 4:44 pm

Post by vivafruit » Sat Jan 13, 2007 9:11 pm

An observation:

Yesterday my Mind Game review was at 9.75 with 5 votes. Today, after someone gave it a 10, the approval rating fell to 9.67. So basically, an additional person giving me a 10 lowered the rating.

This doesn't seem right to me. More people giving a perfect score should always either raise the score or keep it the same, right?

merkhant
Posts: 9
Joined: Wed Jan 10, 2007 11:56 pm

Post by merkhant » Sat Jan 13, 2007 11:50 pm

Well, it seems the new algorithm uses some kind of by-user averaging, probably votes by people who give too many high scores are actually scaled down. So, a 10 from such a person would count as, say 8.7.

No way to be sure, just a user's thought based on seeing through many ratings.

But I agree, that seems unjust.
Wouldn't it be better (if my observations are correct) to change the weight of the votes given by people with high averages (when they are voting high again) instead? For example a 10 from a user with an avarage of 9.5 would count only as a 1/4 vote (but 6 from the same user could have the full weight).
That way a 10 would never lower a score, but could be outweighted by the scores of more moderate users.

Of course that goes also the other way, for very low scores.

Andemon
Posts: 117
Joined: Thu Oct 14, 2004 4:12 pm

Post by Andemon » Mon Jan 15, 2007 12:07 am

Unjust? If you think that's unjust, you don't know the meaning of the word.
Personally, I don't give a damn about how it works. It works better than the old system - that's enough for me.

laxxy
Posts: 20
Joined: Thu Mar 02, 2006 4:34 pm

Post by laxxy » Mon Jan 15, 2007 1:20 am

Well, after a few days it does not look like the new system is much different after all, judging by the top 10 at least :)
I also see 5 animes out of the top 10 tied at 8.67 and three more at 8.69 as a problem, I think the old system provided more variance.
and I think it is wrong when a situation in which *any* vote from a user -- including a 10 -- results in a ratings decline. Its' rather counterintuitive and may discourage regular, honest people from voting, which I am sure no one wants.

nstgc
Posts: 48
Joined: Sun May 22, 2005 6:45 am
Location: Island Closest to Hell

Post by nstgc » Mon Jan 15, 2007 1:45 am

I was just noticing that myself. I was looking through and things seems to be clumping together at 8. While there is nothing wrong with that being the average, it reduces the value of the rating. I haven't looked at all animes, but of those I glanced at, this seems to be the case. In reality it may make little difference, I watch whatever is new regardless of rating, but like the hint system, which I assume is still broken (meaning if nothing has changed in a year), its the principal of a resource being removed needlessly. By needless, I'm not saying that nothing was wrong before hand. Fanboys were ruining the system, no question. I also believe that its still better then how it was, but unless this is an intermediate stage its still a loss as there are other ways of accomplishing the same thing.

laxxy
Posts: 20
Joined: Thu Mar 02, 2006 4:34 pm

Post by laxxy » Mon Jan 15, 2007 2:06 am

nstgc wrote:I also believe that its still better then how it was, but unless this is an intermediate stage its still a loss as there are other ways of accomplishing the same thing.
Perhaps. One thing is that between two systems that produce essentially identical results, I'd always take one that is simpler and more transparent.

nstgc
Posts: 48
Joined: Sun May 22, 2005 6:45 am
Location: Island Closest to Hell

Post by nstgc » Mon Jan 15, 2007 3:30 am

Occam's Razor? Yeah, well without being able to see the mechanics of the new system, we can't judge what is simpler and what is more complex. True, the old system was just about as simple as they get, but take Newtonian physics for instance. Its simpler then Relativistic physics, yet its not more correct. Same may be said about Maxwell's Equations and Quantum Electrodynamics (QED). I am not a strong believer in Occam's Razor.

Think of it this way, atleast they are doing something about it. If the admins are taking note of, and working towards a solution for, the problem of retarded fanboys throughing the votes out of whack, then that indicates progress. Even if we believe its worse, which I don't, atleast its indicating that there is the possibility of future progress. I will admit that with the case of the hint system with which I came up with a nice, simple, three part system that made sense, no progress has been made. It turned into the "find similar" system I believe which uses a, what the guy who was working, "cosine method", which I then showed was actual a fancy way of doing something else; a something else which, for the purpose of suggestions, won't work (it should be fine for a "find similar" system though).

The point is we can only hope that it gets better and that this isn't the final product. In my opinion, its still better. More importantly, it doesn't effect me that much. I never looked at the anime votes, and whenever I was making any serious comparison between subbing groups, I would look at the raw data, whip out OpenCalc (a spread sheat program) and and use my own method for solving the "idiot" issue. So if you have a problem with the system, just do what I do...do it yourself. Thats why I asked once before if in addition to the average, we could be given the second and third moments.

suppy
Posts: 151
Joined: Mon Jan 31, 2005 10:37 am
Location: A House of the Azath

Post by suppy » Mon Jan 15, 2007 4:34 am

nstgc wrote:but like the hint system, which I assume is still broken (meaning if nothing has changed in a year)
Yes, still broken, but what's happened to the ratings seems to be on the table for this as well from what I've heard.
nstgc wrote:but unless this is an intermediate stage its still a loss as there are other ways of accomplishing the same thing.
It is intermediate. Hope it all gets implemented soon. I do. =)

laxxy
Posts: 20
Joined: Thu Mar 02, 2006 4:34 pm

Post by laxxy » Mon Jan 15, 2007 8:15 am

nstgc wrote:Occam's Razor? Yeah, well without being able to see the mechanics of the new system, we can't judge what is simpler and what is more complex. True, the old system was just about as simple as they get, but take Newtonian physics for instance. Its simpler then Relativistic physics, yet its not more correct.
Correct? I didn't say anything about correctness, especially since it is not clear what being 'correct' means in this context. (if resistance to tampering by fanboys were the main criterion, i can suggest an even simpler system: just freeze ratings as of today :) )
But your example is good, as Newtonian physics is still preferred for vast majority of applications, while being technically less correct.

The main problem is the lack of transparency of course. I think that with the help of the community it should be possible to design a system that is open, intuitive and fairly resistant. The main question is the interest on the admin side, and it seems that now the interest is there.
Think of it this way, atleast they are doing something about it.
Definitely, it is nice to see that. I do hope something nice comes out of it.
I will admit that with the case of the hint system with which I came up with a nice, simple, three part system that made sense, no progress has been made. It turned into the "find similar" system I believe which uses a, what the guy who was working, "cosine method", which I then showed was actual a fancy way of doing something else; a something else which, for the purpose of suggestions, won't work (it should be fine for a "find similar" system though).
I recall there being a thread about that, I even made some suggestions there myself, but it seemed the admins were too busy at that time, so it seemed a waste to spend too much time thinking about a system that would perhaps never be considered.
In my opinion, its still better.
Well, other things being equal, I'd probably take a rating with Nausicaa in the top 10 (not for long it seems though) over one without, too :) It does have some funny features, as people noted. One other small fact: a review of mine has 2 votes, "3" and "5", and a current rating of 6.82 :)
More importantly, it doesn't effect me that much.
me neither, although I'd like to have more confidence about how my vote is going to affect the results when voting for a series. Designing a good system is an interesting statistical question though.
Thats why I asked once before if in addition to the average, we could be given the second and third moments.
what do you need the 3rd moments for? :) actually fourth might be more useful to detect fanboys 8)
one thing not to do when looking at such issues is to carry over results that imply a normal or similar distribution to things like votes or rankings.

merkhant
Posts: 9
Joined: Wed Jan 10, 2007 11:56 pm

Post by merkhant » Mon Jan 15, 2007 8:41 am

laxxy wrote:One other small fact: a review of mine has 2 votes, "3" and "5", and a current rating of 6.82 :)
That's the thing I don't like most about the new system. The rating may be totally different from the votes given.
I think we can safely assume that a review that got only votes under 6 should stay disapproved, disregarding avarage vote or fanboyishness of the raters. That goes for other votes too, but is most easily seen with reviews.
That's why I would prefer modifying weights approach.

P.S. Nothing against your review, laxxy, if you PM me a link to it, I'll rate it.

eltimsy
Posts: 1
Joined: Tue Jan 16, 2007 1:58 am

Post by eltimsy » Tue Jan 16, 2007 2:01 am

Just out of curiosity did all the users with multiple accounts get deleted after the new voting system was put in?

Locked