Future of Anime Hint - Anime Referral

Forum for discussing AniDB rules & standards. No small talk!

Moderator: AniDB

nstgc
Posts: 48
Joined: Sun May 22, 2005 6:45 am
Location: Island Closest to Hell

Post by nstgc »

I don't think thats what he's talking about. You rate each anime for their value to one of five genras and then in general. Its not a selection of genras that he's reffering to. You would have to have alot of users recast their votes after having to fit in 5 more feilds. Then that wouldn't fit in with the current algoriths (although I could make them fit), which is no loss since we all know that they don't work anyway. By "make them fit in" I mean mathematicaly, not just writing them into the script. You can code alot of things, but if its just junk then thats all you'll get out of it.
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

nstgc wrote:Thank you very much, I wasn't expecting anything at all untill Saterday. I have questions and comments now though.
I guess it is a good thing I made that post, I haven't had time to do anything since.
egg wrote:Things have progressed since then, but I don’t have any more time right now. Also I want to see if this information is useful before I take the time to do more.
Actually I ended up taking out the other stuff I put in. It was causing more problems than it was helping.
nstgc wrote:Problems:
  • You should subtract by 5.5
  • I don't like how your using an if/then logic for "whose bigger;" it leads to discontinuity and "favortism"
  • You're trying to get rid of the sign, we all know my thoughts on that
I originally had 5.5, but I changed it to 5 for a logical reason (not a mathimatical one). Basically a vote of 5 is supposed to be:
5 - Decent
This anime is neither good nor bad. People can watch it whenever they are bored or when they've run out of other anime to watch.
So it makes sense that anything below this is negative and anything above this should be positive.

Maybe I can explain the logic by giving an example:

Code: Select all

anime u1 u2 u3
----- -- -- --
a1    10 10 10
a2     7  7  7
So, if I compare a1 and a2, the voting has a same trend, that is evident by the cos of the angle (there are pointing the same direction, so the angle is 0 and the cos is 1).
cos = (5*2+5*2+5*2)/(sqrt(5*5+5*5+5*5)*sqrt(2*2+2*2+2*2)) = 30/(sqrt(75)*sqrt(12)) = 30/sqrt(900) = 30/30 = 1

That would mean that we have a perfect score... The only thing is you can tell from the votes that it is NOT a perfect match, so I needed a way of determining how good of a match it really is. What I decided was to multiply the score by the ratio of the norms of the vectors. If all of the votes are the same, then the norms will be the same and it will be a good match, if they are drastically different, then that difference will be reflected in the score. In this case, the norms of the vectors are:
|a1| = sqrt(5*5+5*5+5*5) = sqrt(75) = 5*sqrt(3)
|a2| = sqrt(2*2+2*2+2*2) = sqrt(12) = 2*sqrt(3)

So the score would be:
1 * ((2*sqrt(3))/(5*sqrt(3))) = 1 * (2/5) = 2/5 = 0.4

This turns out not to be a very good match. If there had been all 9s as the votes for a2, then the score would have been much higher (0.8). After going through the numbers, I agree the penalty is probably higher than it should be (I forgot I subtracted 5 before doing the calculation which made difference double what I expected). Maybe I will make an adjustment to lessen the impact of this score. Maybe just using an average of the original votes (instead of the norm) would be better, I'll have to think about this.

As for losing the sign, yes, I am losing the information about if people scored that anime higher or lower than this one. When I first implemented the adjustment using the norms, I would divide by the norm of the current anime and multiply by the other one. That would mean if people voted higher (overall) for the other anime, then the score would be higher. Unfortunately this kept returning highly rated (i.e. top 100) anime in the results for things where they really didn't belong. I also determined, that I wanted the anime where people voted most similarly to this one, not necessarily ones where people voted higher.
nstgc wrote:Will the end product have a core which handles things and then have a bunch of weights and stuff help?
Probably. The first weight is what I tried to use the norm for, but that may have been too drastic. This is a basis that probably needs tweaks to get it to the point where the results are useful. That is why I am making the results public, so that I can get feedback about what works and what appears to be out of whack. If there are any problems, then I can go back look at the data and see how it can be adjusted.
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

sothis wrote:anyways, i do appreciate the desire to not "compete" with my system i have set up... at the moment i'm redesigning and recoding the entire site, including the anirec.

but in any event, exp and i did briefly discuss the algorithm for anidb... i'd be open to the idea of somehow incorporating the data i have with anidb's anime hint, for anidb, at least. in general i'd like to strengthen the link between our sites since they have always been sister sites of sorts. i think we each compliment each other well. anidb for its complciated and accurate algorithm, and anime planet for its 100% user driven recommendations. i think the two give users a very good selection of data to choose from for finding new series :) and i hope with the redesign of anime planet, we can somehow make that bond stronger.
Part of the desire of not competing comes from laziness (I don't want to design an interface to do what you already have). You have a wonderful site and I never intended to try to compete, but there is a wealth of information here that could be tapped into and hopefully find parallels that are otherwise overlooked. Having a merged system though appears to be a popular idea, so if you and exp work something out that would be great.
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

nstgc wrote:I hate double posting, but I will in this case. I want this topic to be flagged as new message.

Please read the fallowing pdf file.

[url]http://nst_gc.tripod.com/cosine_hints.pdf[/url]
OK, interesting way of dealing with the if... It works, it is computationally expensive, but it works.

I basically can follow what you said in the document, although in some cases I am missing something. Since I don't fully understand where you are going with the equations, I do not see the issue that you a trying to explain. I am more of a numbers man ran than theoretical. If you can create a sample (small sample) that exhibits your concern, then I think it will better understand it and the actual number should speak for itself... As you can tell from my other post, I know the ratio of the norms is not perfect, but it is what I came up with at the time. Adding it in appeared to have the affect that I wanted, so I kept it...

[I know you don't like multiple posts, but I had to reply to things in order to keep my sanity.]
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

Rar wrote:Well, more to the point, when categories are shiny, integrating them to the hint (rather than the current genre hack) should actually be quite painless... If you want sci-fi-high results, you could get 'em. I don't think they should be the *basis* for rec suitability though.

Rar
So, you are saying that categories should be used for filters, and not necessarily for determining the animes score (i.e. used in the equations)?

Well, let me put out what I am thinking right now.
1) For every anime there will be a list of "similar" anime. This will be determined primarily by the categories (and as long as there are genres, this information will be folded into the categories as well). Votes, Year, Release Type, Number of Eps, and other things may also go into the score. This list does not filter and does not use any user information.
2) For every anime there may be a list of anime "referrals". This is what I am working on currently. This is primarily based on the Votes. Categories, year, release type, number of eps, and other things may also go into the score. This list does not filter and does not use any user information.
3) There will be a hint added that will basically go to the anime that you voted on and it will use these lists (one or the other or both) and your votes to determine a suitable hint list. The hint will have all the same filters that are available to the other hints (although at some point genres will be replaced by categories).

Now in the end it may turn out that 1 and 2 become the same list, but right now I am assuming they are different.
nstgc
Posts: 48
Joined: Sun May 22, 2005 6:45 am
Location: Island Closest to Hell

Post by nstgc »

The thing with the "subtract 5 instead of 5.5" is that the equation uses squares, and if your data is off balance it will cause problems. Not that it really matters.

What that last part was doing was just showing you that this cosines method is the same corrilation method at heart.

The middle part shows that your results will be thrown off becuase your taking the quadratic mean which turns out to be influence by the spread.

The reason I hate double posting is that it reminds me of n00bs, and I hate n00bs. In your case it makes things nice and orderly. So when other people make multiple posts it doesn't bother me unless their being stupid. If I'm the one making the multiple post then it bothers me a whole lot.

As mentioned before, I hate making examples. I'll try to make an example, but I don't think I can. If you need a better explination I'll do that. The point I'm making is that it will be off. To show that it will be off with an example you need to be able to compare it against something and that something doesn't exist. The problem can only be expressed properly in the manner that I have provided.

Also, that first equation, was that basicly how the system works? It doesn't really matter much now since as long as there is that ratio its wrong, but I would like to make sure we're on the same page.
Guest

Post by Guest »

nstgc wrote:The thing with the "subtract 5 instead of 5.5" is that the equation uses squares, and if your data is off balance it will cause problems. Not that it really matters.
The data is basically off balance because people tend to vote with higher votes overall than lower votes anyway... I don't think that this will have a major impact on the outcome anyway.
nstgc wrote:What that last part was doing was just showing you that this cosines method is the same corrilation method at heart.
I agree that they are similar.
nstgc wrote:The middle part shows that your results will be thrown off becuase your taking the quadratic mean which turns out to be influence by the spread.
OK, I got that, but I'm not sure that would make the result any less value. Anyway, I don't think this is a major concern because if there is a large spread, then the cosine result will probably be fairly low anyway and that value will be filtered out.
nstgc wrote:As mentioned before, I hate making examples. I'll try to make an example, but I don't think I can. If you need a better explination I'll do that. The point I'm making is that it will be off. To show that it will be off with an example you need to be able to compare it against something and that something doesn't exist. The problem can only be expressed properly in the manner that I have provided.
OK, how about I give you an example and you give me a method to resolve the issue.

Let us take the following scenario, there are 6 animes a1-a6 and 3 users, u1-u3 (the number of users is not really important, I just chose three so that there are not that many users to deal with, if you need more, add more). All of the users vote exactly the same. We are trying to find the the anime with the closest voting pattern to a1.

Code: Select all

Anime u1 u2 u3
----- -- -- --
a1     8  8  8
a2    10 10 10
a3     9  9  9
a4     8  8  8
a5     7  7  7
a6     6  6  6
With the cosine, all of these animes would have a perfect score (as demonstrated in my previous example). But intuitively a4 should have a better score than the other because it is an EXACT match. I also believe that a4 and a3 should be close, a2 and a6 a little farther away. With the current system, these would have the following values: a2 = 600, a3 = 750, a4 = 1000, a5 = 666, a5 = 333. If I hadn't subtracted 5 from the values first, it would not have been that big of a difference... I think that current system needs to be changed, but the question is how...
nstgc wrote:Also, that first equation, was that basicly how the system works? It doesn't really matter much now since as long as there is that ratio its wrong, but I would like to make sure we're on the same page.
Yes, my explanation should be the way it works.

I just had an idea. Another issue with the current system is that it depends on votes that it does not quite fit what I am looking for at the moment. What I am calculating (ideally, barring errors with logic) is how similarly people voted between two animes. Although this is useful, it is not quite what I want. What I want is to know if someone likes this anime, what other animes will they like. Well, with the current system, lets say a number of users vote 1s and 2s for two anime. Their voting is similar and would result in a high score, but that does not necessarily mean that someone who liked one of the anime would like the other one. This is something that I have been thinking about for a while, I had implemented one idea in a previous trial, but that turned out to cause other problems. So, I just had a thought. Since I want to know how people who like this amine will like, maybe I should just look at the users who like or recommend the anime (or a vote of 7 or more). Lower votes for this anime would get filtered out. If someone votes 7 for this anime, but 1 for another anime, that one will remain... So let's look at this set of votes:

Code: Select all

Anime u1 u2 u3 u4 u5 u6 u7 u8 u9
----- -- -- -- -- -- -- -- -- --
a1     1  7  8  5  9  8  3  7  9
a2    10  6 10  5  9  6  1  5  8
When comparing a1 to a2, the following votes would be used:

Code: Select all

Anime u2 u3 u5 u6 u8 u9
----- -- -- -- -- -- --
a1     7  8  9  8  7  9
a2     6 10  9  6  5  8
But when comparing a2 to a1, the following votes would be used:

Code: Select all

Anime u1 u3 u5 u9
----- -- -- -- --
a1     1  8  9  9
a2    10 10  9  8
That would mean that a1's referral score for a2 would be different than a2's referral score for a1.

Any thoughts? The other logic will not go away, it will probably be used for finding similar animes and/or averaged somehow with the new result. I will probably mock this up to see what happens. I know this will sigificantly reduce the data set, but if the other data is just noise, then that shouldn't be a problem.
fahrenheit
AniDB Staff
Posts: 438
Joined: Thu Apr 08, 2004 1:43 am
Location: Portugal

Post by fahrenheit »

i have been thinking on the subject some more,

okay, imagine we have this:

Code: Select all

____|_U1_|_U2_|_U3_|_U4_|
_A1_|__8_|__2_|__1_|__7_|
_A2_|__7_|__3_|__8_|__7_|
_A3_|__8_|__4_|__7_|__6_|
_A4_|__9_|__5_|_10_|__8_|
_A5_|__6_|__4_|__2_|__8_|

_A6_|__X_|__5_|__7_|__6_|
we try to guess what user1 would vote for anime6, my ideia is to do the following:
1) get a small sample (10 or so) of users which have voted for the same animes as user1, and are close matches in votes for each (a 0.5/10 deviation at the max);
2) try to extrapolate what the user would have voted for anime6.
3) if the extrapolated vote is above a given value, for example if the average vote of u1 is 5.6, and the computed vote is 63% above of the average deviation of votes + the average vote (say 5.6 + 0.63 * 3.28 = 7.7 for user3), add that anime to the recommend animes.

i have tried to do some math with a calc sheet, but i didn't get to any conclusion, my statistics are very poor..

does this help anything?
nstgc
Posts: 48
Joined: Sun May 22, 2005 6:45 am
Location: Island Closest to Hell

Post by nstgc »

egg wrote:
nstgc wrote:The thing with the "subtract 5 instead of 5.5" is that the equation uses squares, and if your data is off balance it will cause problems. Not that it really matters.
The data is basically off balance because people tend to vote with higher votes overall than lower votes anyway... I don't think that this will have a major impact on the outcome anyway.
I know its skewed. I have my meen at 7.2 or something. If you want to make things balanced you would standarize. Also, 5.5 is a more logical number then 5. Since I am pushing for the removal of the system altogether I won't bother trying to explain why.
egg wrote:
nstgc wrote:The middle part shows that your results will be thrown off becuase your taking the quadratic mean which turns out to be influence by the spread.
OK, I got that, but I'm not sure that would make the result any less value. Anyway, I don't think this is a major concern because if there is a large spread, then the cosine result will probably be fairly low anyway and that value will be filtered out.
While not the type of example that you seem to like, but this should be effective.

Consier two animes. The number of votes is irrelevant also I'm not substracting 5 since it doesn't matter for this example. Lets say the first has users that vote only 5's. They would have a standard deviate of 0 and an average vote of 5. The second anime has users who vote all over the place. They may have a STD of 5 and still have an average of 5. The ratio ends up as 1/2. You could say that the validity is lessed becuase of the inceased spread, so the score is lessened. Well what if the first one had been 1 as well, then the validity would be even lower, yet the score is higher. The standard diviation is a mesure of how much how much of the result is within a certain distance of the mean. You use it in the same way I use it in my weighting system, to show accuracy or validity. The mean is a magnitude. It is how much most people voted. That is not the way to put those two together. Its not that they don't go together, its just that its not in that manner.
egg wrote:
nstgc wrote:As mentioned before, I hate making examples. I'll try to make an example, but I don't think I can. If you need a better explination I'll do that. The point I'm making is that it will be off. To show that it will be off with an example you need to be able to compare it against something and that something doesn't exist. The problem can only be expressed properly in the manner that I have provided.
OK, how about I give you an example and you give me a method to resolve the issue.

Let us take the following scenario, there are 6 animes a1-a6 and 3 users, u1-u3 (the number of users is not really important, I just chose three so that there are not that many users to deal with, if you need more, add more). All of the users vote exactly the same. We are trying to find the the anime with the closest voting pattern to a1.

Code: Select all

Anime u1 u2 u3
----- -- -- --
a1     8  8  8
a2    10 10 10
a3     9  9  9
a4     8  8  8
a5     7  7  7
a6     6  6  6
With the cosine, all of these animes would have a perfect score (as demonstrated in my previous example). But intuitively a4 should have a better score than the other because it is an EXACT match. I also believe that a4 and a3 should be close, a2 and a6 a little farther away. With the current system, these would have the following values: a2 = 600, a3 = 750, a4 = 1000, a5 = 666, a5 = 333. If I hadn't subtracted 5 from the values first, it would not have been that big of a difference...
In that part you are quoting, I was talking about something else. I understand perfectly how the scores would look with and without your standardizing.
egg wrote: I think that current system needs to be changed, but the question is how...
If you are talking about how to standardize, I'm working on that and currently have some ideas, but I don't full confidence in any of them at this time. I'll need to look at them a little more before deeming useful and correct (those two don't have to come as a pair). If you are are speaking about the system as a whole, then I've been pointing out problems and explaining why they are a problem, while often times suggesting solutions and I even made a a whole new algorithem. In either case you should not feel like your out of options since in the first case all you need to do is ask, and in the second case I've said more then I should.
egg wrote:
nstgc wrote:Also, that first equation, was that basicly how the system works? It doesn't really matter much now since as long as there is that ratio its wrong, but I would like to make sure we're on the same page.
Yes, my explanation should be the way it works.

I just had an idea. Another issue with the current system is that it depends on votes that it does not quite fit what I am looking for at the moment. What I am calculating (ideally, barring errors with logic) is how similarly people voted between two animes. Although this is useful, it is not quite what I want. What I want is to know if someone likes this anime, what other animes will they like. Well, with the current system, lets say a number of users vote 1s and 2s for two anime. Their voting is similar and would result in a high score, but that does not necessarily mean that someone who liked one of the anime would like the other one. This is something that I have been thinking about for a while, I had implemented one idea in a previous trial, but that turned out to cause other problems. So, I just had a thought. Since I want to know how people who like this amine will like, maybe I should just look at the users who like or recommend the anime (or a vote of 7 or more). Lower votes for this anime would get filtered out. If someone votes 7 for this anime, but 1 for another anime, that one will remain... So let's look at this set of votes:

Code: Select all

Anime u1 u2 u3 u4 u5 u6 u7 u8 u9
----- -- -- -- -- -- -- -- -- --
a1     1  7  8  5  9  8  3  7  9
a2    10  6 10  5  9  6  1  5  8
When comparing a1 to a2, the following votes would be used:

Code: Select all

Anime u2 u3 u5 u6 u8 u9
----- -- -- -- -- -- --
a1     7  8  9  8  7  9
a2     6 10  9  6  5  8
But when comparing a2 to a1, the following votes would be used:

Code: Select all

Anime u1 u3 u5 u9
----- -- -- -- --
a1     1  8  9  9
a2    10 10  9  8
That would mean that a1's referral score for a2 would be different than a2's referral score for a1.

Any thoughts? The other logic will not go away, it will probably be used for finding similar animes and/or averaged somehow with the new result. I will probably mock this up to see what happens. I know this will sigificantly reduce the data set, but if the other data is just noise, then that shouldn't be a problem.
Why did you ommit two users?

I do know what you're saying before your example. I also know that you throwing out useres just becuase they don't like something is an absolutely horrible idea. If you hate negitive scores that much I'll rig something up to fix something that isn't broken if you want, but you should realize that whimsicaly discarding information is irrisponsible. Unless the data is erred you work with the data and mildly inconvieniant data is not caused by an error. Mildly inconvient data is cuased by infelixblity.
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

OK, a new version is available here. This just does the cosine and applies the ratio (the way nstgc doesn't like). This is based on different sets of data, you can sort by any of the columns and see how the results vary by which data it is using. The question is which set (or sets) of data appear to reflect the results that we want. Remember the goal is, if someone voted highly for the anime you are viewing, this should be a list of animes that this person would like (based on voting patterns), not necessarily the most similar anime (in content).

BTW, I will respond to the other messages later.
nstgc
Posts: 48
Joined: Sun May 22, 2005 6:45 am
Location: Island Closest to Hell

Post by nstgc »

Moving away from the errous method that has missing data, I have a method of standardizing the data along multiple axies (in this case anime and users) for the dependent variable (votes). That was the extremely easy part. Now I have to make sure it isn't a load of crap (I perffer to have everything I tell you mathematicaly correct as well as logical). It is already logical, but I need to ensure that it is best for the data.
laxxy
Posts: 20
Joined: Thu Mar 02, 2006 4:34 pm

Post by laxxy »

I haven't read the whole thread yet, but this is all very interesting.

Since there was some interest, here's an outline possible way of finding recommendations that does not directly rely upon finding similar users.
I'll describe a very simple version, with some notes on how it could be improved further.

1. Compute an average rating for each anime To start, one could take the average vote rating that we have on the front page. (In general imo it should be improved, at least de-meaning the votes of each user, but it does not have to be very precise).

2. Create variables that specify some groups of anime's. For a start, simple genre dummies should suffice (e.g. an action dummy is a variable that is equal to 1 if it's action, 0 otherwise; we may also do some correction for animes that span multiple genres). In principle cluster analysis would be one other potentially good thing to try here.

3. For a given user, run an OLS (aka linear) regression of the (log of?) his vote on:
-log of the anime score
-genre dummies (in future possibly replaced by cluster distances)
-possibly, other variables: the year anime was done in, number of episodes, etc.
There has to be a fair amount of trial and error here to find good (i.e. significant for many people) variables. In some cases logs or level dummies might work better than actual values, etc.
Later, more appropriate methods could replace OLS too.

4. Using the coefficients from step 3, rank the animes by predicted values.
nstgc
Posts: 48
Joined: Sun May 22, 2005 6:45 am
Location: Island Closest to Hell

Post by nstgc »

I would like to comment on the above aside from "you have as much chance as I do," but thats too depressing so I won't. I don't really fallow and I don't really care (something in abundance around here). But really, it does seem like it was thought out to some degree, unfortunately any new system suggested is trashed. I'm about to give an update on my own system, but I already know its pointless.

I have recently, a few hours ago, found a second interpretation of my CRT. The first was intrepreted as:
The difference between the vote of user i on anime 0 and the off set of user i and 0 produces an estimated vote for user 0 on anime 0. When averaged over N users in repects to i, this estimator becomes more accurate. As N tends towards infnity, this estimator converges onto a final estimator for that given set of data.
From this I came to beleive that this "final estimator" was perfect. With in this intrepretation, I have no reason to beleive otherwise.

The second, and almost as correct, intrepration is:
The average of the difference between the vote of user i on anime 0 and that users average vote produces aproximated difference between any user j's vote on anime 0 and that users average vote. When user 0 adds his average vote to the previous result, an estimated vote for user 0 on anime 0 is formed. As the number of users this average difference encompases increases to infinity, the average will settle onto a final result.
If a user average vote isn't "correct" then I believe the result converges onto a wrong answer. There are some differences in the two formulas that these interpretations come from, but the difference is very small and there is no reason for it to alter the result and it doesn't change the interpretation at all.

I am of the opinion that when a person realizes they've made a mistake, they point it out themselves. I'm not sure I'm wrong yet, but I have two equivilant formulas that are either both right or both wrong, yet the two disagree.

On the plus side, using the new intpretation, I can further develope the CRT and make it converge with fewer users and without weights. Also, I should be able to "invert" the formula. Its easy to do, but getting a meaningful interpretation is hard. I hope I can do a better job now.

I wanted to say something else, but I forgot what it was.
Locked