re-using kowai

Want to help out? Need help accessing the AniDB API? This is the place to ask questions.

Moderator: AniDB

User avatar
Rar
AniDB Staff
Posts: 1471
Joined: Fri Mar 12, 2004 2:41 pm
Location: UK
Contact:

Post by Rar » Fri Jun 01, 2007 11:40 pm

Meh, someone give me a reason not to trash this whole thread. Whole thing is just a mess of half-wrongs that'll confuse anyone who actually wants useful answers on anything related to making a decent client.
What's with this use of the word gestalt? Its completely inconsistant with any of the established meanings in either german or english - some fringe ITbabble term? Seriously, if you start talking about monads next I'll petition to ban you for perversion.
At any rate, keeping a synced copy of any of the large parts of anidb was what I was refering to as "essentially not practical", because #1 you can't re-get everything every time anything changes, #2 there is no commit log, #3 you can't limit changes to only the routes you control (Chii can make changes to mylists that the UDP api will know nothing about, for instance).
All this talk about scraping mylist pages is crap though, when you can just get a clean dump in one of several flavours.
I do love breaking scrapers though...

Rar

User avatar
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Post by exp » Sat Jun 02, 2007 9:15 am

well,

I guess there is one point in this discussion.
That is, mylist exports being not quite easy to use from a client perspective.
The main problem there would be the delay between export request and availability of the export.

Maybe it might be usefull to add one more channel to retrieve mylist information from. Something which is xml based and works without delay. If we carefully decide on the amount of data in such an xml representation of a user's mylist, generating it would actually be less work for the server than a generating a full mylist export.
And it beats tons of html scrapping requests anytime.

BYe!
EXP

titoum
Posts: 29
Joined: Mon Sep 13, 2004 10:48 am
Location: Plop

Post by titoum » Sat Jun 02, 2007 10:59 am

i was just wondering, wouldn't it be possible to display the mylist page as an xml ?

so we would have juste to save it ? dunno

like the gentoo.org website

User avatar
Rar
AniDB Staff
Posts: 1471
Joined: Fri Mar 12, 2004 2:41 pm
Location: UK
Contact:

Post by Rar » Sat Jun 02, 2007 2:03 pm

We do, look at the link I gave. You're *still* not going to be able to keep a good local copy of it though.

Rar

MostAwesomeDude
Posts: 38
Joined: Fri Jun 01, 2007 11:02 am

Post by MostAwesomeDude » Sun Jun 03, 2007 8:43 pm

"Gestalt" is the word used at my work for any type of binary snapshot. I blame it on my boss, who's like 65 and fading fast. He talks about monads, too, actually. Rambles on for hours on end. He pays well, though.

Yeah, caching a database ain't practical, but it certainly beats dumping the database, right? Kowai is a series of sqlite tables, encrypted with a private key. As much as I <sarcasm>absolutely love</sarcasm> wading through disassembled memory dumps, it's simply not my thing. I prefer to write code, not reverse code.

At any rate, keeping a synced copy of any of the large parts of anidb was what I was refering to as "essentially not practical", because #1 you can't re-get everything every time anything changes, #2 there is no commit log, #3 you can't limit changes to only the routes you control (Chii can make changes to mylists that the UDP api will know nothing about, for instance).

As far as cache practicality, you simply have to have two abilities: You need to be able to correctly handle unexpected database return codes, and you need to be able to invalidate any part of the cache at any time. Combining these two, you can create a cached database that only mirrors and updates itself in accordance with what is requested of it. After all, the UDP API was not designed to mirror the database, just small parts.

The three conditions are: something missing from your cache, something in your cache that is not in the DB, and something in your cache that does not match the DB. The cache is already initially set up to handle #1, and #2 is identified through status codes like NO SUCH ANIME. #3 conditions can be handled on-the-fly as they occur through user requests.

For example, let us say that the mylist is cached, that the cache is good, and that the number of files added to the mylist through the client is stored in the cache. The client caches each MYLISTADD and remembers the files in its cache. All it has to do to check the cache's consistency is call at most one MYLISTSTATS and one MYLIST, and it will know the condition of its cache.

Code: Select all

call MYLISTSTATS
is # of files from MYLISTSTATS equal to # of files in mylist cache?
yes:
    call MYLIST lid=(last good lid in cache + 1)
    is MYLIST status 321?
    yes:
        # Cache is valid!
    no:
        # Cache is invalid!
no:
    # Cache is invalid!
With a bit more condition-checking and one more (or is it two?) MYLIST call, it is possible to identify exactly which race condition occurred, and the correct algorithm to try. (Proof is left to the reader, on account of it takes a lot of pseudocode and I don't feel like typing it all out again.)

The mylist is the most important thing to be able to cache and keep consistent, since it is important to the user and also the quickest-changing part of the DB.

~ C.

User avatar
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Post by exp » Sun Jun 03, 2007 11:05 pm

without going into the rest of your post, one important note:
mylist ids are global, by incrementing your last lid you will most likely end up with the lid of another user and will therefore only get a 502 ACCESS DENIED reply.

BYe!
EXP

MostAwesomeDude
Posts: 38
Joined: Fri Jun 01, 2007 11:02 am

Post by MostAwesomeDude » Mon Jun 04, 2007 6:08 am

exp wrote:without going into the rest of your post, one important note:
mylist ids are global, by incrementing your last lid you will most likely end up with the lid of another user and will therefore only get a 502 ACCESS DENIED reply.

BYe!
EXP
*blink* Well, that explains this afternoon, then. Wish that was in the docs...

Back to the drawing board! *maniacal cough*

~ C.

yurko savelenko
Posts: 10
Joined: Thu May 03, 2007 4:43 pm

Post by yurko savelenko » Sat Jun 09, 2007 10:14 pm

It actually is stated in the documentation, just study carefully the wiki page on UDP protocol.

User avatar
fahrenheit
AniDB Staff
Posts: 438
Joined: Thu Apr 08, 2004 1:43 am
Location: Portugal

Post by fahrenheit » Sun Jun 10, 2007 12:12 am

yurko savelenko wrote:It actually is stated in the documentation, just study carefully the wiki page on UDP protocol.
now, is stated on the wiki page, he added it :P

yurko savelenko
Posts: 10
Joined: Thu May 03, 2007 4:43 pm

Post by yurko savelenko » Fri Jun 15, 2007 1:31 pm

Who is "he"? I did not add anything. It has always been there.

User avatar
fahrenheit
AniDB Staff
Posts: 438
Joined: Thu Apr 08, 2004 1:43 am
Location: Portugal

Post by fahrenheit » Fri Jun 15, 2007 1:52 pm

yurko savelenko wrote:Who is "he"? I did not add anything. It has always been there.
http://wiki.anidb.info/wiki/index.php?t ... oldid=7438

Someone, i'm guessing he ("MostAwesomeDude") added that line to the wiki:
+ * Mylist IDs (lid) are globally unique, not per-user unique.
on 04.06.2007, if you are not talking about that, forget it.

Locked