New rating calculations/system
Moderator: AniDB
-
ohtoriakio
- Posts: 11
- Joined: Fri Apr 28, 2006 10:45 pm
Dunno if this is due to the new rating system or something else altogether but a couple of votes on my reviews seem to have been "ghosted". They are there when I look at the feedback I received but they aren't on my main review page. This also caused one of my reviews that reached the 5 votes mark to not make it out of the "Unrated" section, is there something I can do about that? Thanks for your time.
laxxy wrote:Speaking about groups, I think there is definitely a problem: you've got the first place occupied by WF/KAA, who made all of 4 episodes of one series together before quitting. There needs to be some adjustment for the # of votes, or # of episodes subbed, or smth like that.pelican wrote:Yes, we could tell you. No, we're not going to. (And KAA happen to be pretty good among dvd rippers, which admittedly isn't saying much. Recent KAA anyway.)
that was already fixed in the next version which is not live yet
An observation:
Yesterday my Mind Game review was at 9.75 with 5 votes. Today, after someone gave it a 10, the approval rating fell to 9.67. So basically, an additional person giving me a 10 lowered the rating.
This doesn't seem right to me. More people giving a perfect score should always either raise the score or keep it the same, right?
Yesterday my Mind Game review was at 9.75 with 5 votes. Today, after someone gave it a 10, the approval rating fell to 9.67. So basically, an additional person giving me a 10 lowered the rating.
This doesn't seem right to me. More people giving a perfect score should always either raise the score or keep it the same, right?
Well, it seems the new algorithm uses some kind of by-user averaging, probably votes by people who give too many high scores are actually scaled down. So, a 10 from such a person would count as, say 8.7.
No way to be sure, just a user's thought based on seeing through many ratings.
But I agree, that seems unjust.
Wouldn't it be better (if my observations are correct) to change the weight of the votes given by people with high averages (when they are voting high again) instead? For example a 10 from a user with an avarage of 9.5 would count only as a 1/4 vote (but 6 from the same user could have the full weight).
That way a 10 would never lower a score, but could be outweighted by the scores of more moderate users.
Of course that goes also the other way, for very low scores.
No way to be sure, just a user's thought based on seeing through many ratings.
But I agree, that seems unjust.
Wouldn't it be better (if my observations are correct) to change the weight of the votes given by people with high averages (when they are voting high again) instead? For example a 10 from a user with an avarage of 9.5 would count only as a 1/4 vote (but 6 from the same user could have the full weight).
That way a 10 would never lower a score, but could be outweighted by the scores of more moderate users.
Of course that goes also the other way, for very low scores.
Well, after a few days it does not look like the new system is much different after all, judging by the top 10 at least 
I also see 5 animes out of the top 10 tied at 8.67 and three more at 8.69 as a problem, I think the old system provided more variance.
and I think it is wrong when a situation in which *any* vote from a user -- including a 10 -- results in a ratings decline. Its' rather counterintuitive and may discourage regular, honest people from voting, which I am sure no one wants.
I also see 5 animes out of the top 10 tied at 8.67 and three more at 8.69 as a problem, I think the old system provided more variance.
and I think it is wrong when a situation in which *any* vote from a user -- including a 10 -- results in a ratings decline. Its' rather counterintuitive and may discourage regular, honest people from voting, which I am sure no one wants.
I was just noticing that myself. I was looking through and things seems to be clumping together at 8. While there is nothing wrong with that being the average, it reduces the value of the rating. I haven't looked at all animes, but of those I glanced at, this seems to be the case. In reality it may make little difference, I watch whatever is new regardless of rating, but like the hint system, which I assume is still broken (meaning if nothing has changed in a year), its the principal of a resource being removed needlessly. By needless, I'm not saying that nothing was wrong before hand. Fanboys were ruining the system, no question. I also believe that its still better then how it was, but unless this is an intermediate stage its still a loss as there are other ways of accomplishing the same thing.
Perhaps. One thing is that between two systems that produce essentially identical results, I'd always take one that is simpler and more transparent.nstgc wrote:I also believe that its still better then how it was, but unless this is an intermediate stage its still a loss as there are other ways of accomplishing the same thing.
Occam's Razor? Yeah, well without being able to see the mechanics of the new system, we can't judge what is simpler and what is more complex. True, the old system was just about as simple as they get, but take Newtonian physics for instance. Its simpler then Relativistic physics, yet its not more correct. Same may be said about Maxwell's Equations and Quantum Electrodynamics (QED). I am not a strong believer in Occam's Razor.
Think of it this way, atleast they are doing something about it. If the admins are taking note of, and working towards a solution for, the problem of retarded fanboys throughing the votes out of whack, then that indicates progress. Even if we believe its worse, which I don't, atleast its indicating that there is the possibility of future progress. I will admit that with the case of the hint system with which I came up with a nice, simple, three part system that made sense, no progress has been made. It turned into the "find similar" system I believe which uses a, what the guy who was working, "cosine method", which I then showed was actual a fancy way of doing something else; a something else which, for the purpose of suggestions, won't work (it should be fine for a "find similar" system though).
The point is we can only hope that it gets better and that this isn't the final product. In my opinion, its still better. More importantly, it doesn't effect me that much. I never looked at the anime votes, and whenever I was making any serious comparison between subbing groups, I would look at the raw data, whip out OpenCalc (a spread sheat program) and and use my own method for solving the "idiot" issue. So if you have a problem with the system, just do what I do...do it yourself. Thats why I asked once before if in addition to the average, we could be given the second and third moments.
Think of it this way, atleast they are doing something about it. If the admins are taking note of, and working towards a solution for, the problem of retarded fanboys throughing the votes out of whack, then that indicates progress. Even if we believe its worse, which I don't, atleast its indicating that there is the possibility of future progress. I will admit that with the case of the hint system with which I came up with a nice, simple, three part system that made sense, no progress has been made. It turned into the "find similar" system I believe which uses a, what the guy who was working, "cosine method", which I then showed was actual a fancy way of doing something else; a something else which, for the purpose of suggestions, won't work (it should be fine for a "find similar" system though).
The point is we can only hope that it gets better and that this isn't the final product. In my opinion, its still better. More importantly, it doesn't effect me that much. I never looked at the anime votes, and whenever I was making any serious comparison between subbing groups, I would look at the raw data, whip out OpenCalc (a spread sheat program) and and use my own method for solving the "idiot" issue. So if you have a problem with the system, just do what I do...do it yourself. Thats why I asked once before if in addition to the average, we could be given the second and third moments.
Yes, still broken, but what's happened to the ratings seems to be on the table for this as well from what I've heard.nstgc wrote:but like the hint system, which I assume is still broken (meaning if nothing has changed in a year)
It is intermediate. Hope it all gets implemented soon. I do. =)nstgc wrote:but unless this is an intermediate stage its still a loss as there are other ways of accomplishing the same thing.
Correct? I didn't say anything about correctness, especially since it is not clear what being 'correct' means in this context. (if resistance to tampering by fanboys were the main criterion, i can suggest an even simpler system: just freeze ratings as of todaynstgc wrote:Occam's Razor? Yeah, well without being able to see the mechanics of the new system, we can't judge what is simpler and what is more complex. True, the old system was just about as simple as they get, but take Newtonian physics for instance. Its simpler then Relativistic physics, yet its not more correct.
)But your example is good, as Newtonian physics is still preferred for vast majority of applications, while being technically less correct.
The main problem is the lack of transparency of course. I think that with the help of the community it should be possible to design a system that is open, intuitive and fairly resistant. The main question is the interest on the admin side, and it seems that now the interest is there.
Definitely, it is nice to see that. I do hope something nice comes out of it.Think of it this way, atleast they are doing something about it.
I recall there being a thread about that, I even made some suggestions there myself, but it seemed the admins were too busy at that time, so it seemed a waste to spend too much time thinking about a system that would perhaps never be considered.I will admit that with the case of the hint system with which I came up with a nice, simple, three part system that made sense, no progress has been made. It turned into the "find similar" system I believe which uses a, what the guy who was working, "cosine method", which I then showed was actual a fancy way of doing something else; a something else which, for the purpose of suggestions, won't work (it should be fine for a "find similar" system though).
Well, other things being equal, I'd probably take a rating with Nausicaa in the top 10 (not for long it seems though) over one without, tooIn my opinion, its still better.
It does have some funny features, as people noted. One other small fact: a review of mine has 2 votes, "3" and "5", and a current rating of 6.82 me neither, although I'd like to have more confidence about how my vote is going to affect the results when voting for a series. Designing a good system is an interesting statistical question though.More importantly, it doesn't effect me that much.
what do you need the 3rd moments for?Thats why I asked once before if in addition to the average, we could be given the second and third moments.
actually fourth might be more useful to detect fanboys 
one thing not to do when looking at such issues is to carry over results that imply a normal or similar distribution to things like votes or rankings.
That's the thing I don't like most about the new system. The rating may be totally different from the votes given.laxxy wrote:One other small fact: a review of mine has 2 votes, "3" and "5", and a current rating of 6.82
I think we can safely assume that a review that got only votes under 6 should stay disapproved, disregarding avarage vote or fanboyishness of the raters. That goes for other votes too, but is most easily seen with reviews.
That's why I would prefer modifying weights approach.
P.S. Nothing against your review, laxxy, if you PM me a link to it, I'll rate it.