API: Update multiple items with a single request.

AniDB Client Feature Requests go here

Moderators: AniDB, AniDB API

Locked
Elberet
Posts: 778
Joined: Sat Jul 19, 2003 8:14 pm

API: Update multiple items with a single request.

Post by Elberet » Wed Jul 07, 2004 4:48 pm

In an effort to fix my recently messed up mylist (*glares at PetriW*), I'm setting a lot of files back to "watched", and according to the log AOM actually updates each file individually, at a pace of six or seven files per second (limited most likely by my ping time to AniDB). If it were possible to submit such requests in a block, the whole procedure would be much faster. You could give EBML a try (Extensible Binary Markup Language), the thing Matroska uses to organize an MKV file's headers.

By the way... with "much faster" I mean "a couple of seconds" as opposed to - currently - "tens of minutes". Besides, I can't believe that processing so many singular requests is very friendly on AniDB's database backend. -_-;

taltamir
Posts: 36
Joined: Wed Mar 23, 2005 4:53 am

Post by taltamir » Wed Mar 23, 2005 7:35 pm

the aom auto update might be bugged, but not the manual update as far as I can tell...

Go to mylist IN aom, shift select all the series/eps you want to set to watched, right click, and select set to watch/unwatched/onHDD or whatever suits your fancy.

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Thu Mar 24, 2005 12:50 pm

You don't understand the thead so your reply is not even related to the problem in question.


And Elberet, it's an api limitation. Nothing aom can do about it really, except flood the db with changes and I'm not sure that'd be better.
The real problem is that each command is a separate transaction.

Devil Doll
Posts: 49
Joined: Sat Mar 26, 2005 1:29 pm

Post by Devil Doll » Sun Aug 07, 2005 1:33 pm

PetriW wrote:And Elberet, it's an api limitation. Nothing aom can do about it really, except flood the db with changes and I'm not sure that'd be better.
The real problem is that each command is a separate transaction.
But isn't that a design problem of the AOM client?

Look, if I use the HTML interface to the mylist of aniDB I can select the checkboxes of many entries and then apply a command to it, such as "mark seen". This will be sent as one HTTP POST request to the HTTP server and be performed as one transaction by aniDB. (If it weren't a POST request I'd even give you the exact URL here.)
So I believe it would be the task of the AOM client to be aware of the uniformity of requests and bundle them together (with some limit as not to overrun the database, like no more than 50 files at a time). This way you might be able to significantly reduce the communication overhead, and even partially release the aniDB server of HTTP load.

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Sun Aug 07, 2005 3:20 pm

Devil Doll wrote:
PetriW wrote:And Elberet, it's an api limitation. Nothing aom can do about it really, except flood the db with changes and I'm not sure that'd be better.
The real problem is that each command is a separate transaction.
But isn't that a design problem of the AOM client?
I see you're on a crusade today...

The client will try to group commands. What you ask for you don't understand I think. What you ask for is that I group commands for 15 minutes, then run them.
Devil Doll wrote:Look, if I use the HTML interface to the mylist of aniDB I can select the checkboxes of many entries and then apply a command to it, such as "mark seen". This will be sent as one HTTP POST request to the HTTP server and be performed as one transaction by aniDB. (If it weren't a POST request I'd even give you the exact URL here.)
So I believe it would be the task of the AOM client to be aware of the uniformity of requests and bundle them together (with some limit as not to overrun the database, like no more than 50 files at a time). This way you might be able to significantly reduce the communication overhead, and even partially release the aniDB server of HTTP load.
Aom is already far more efficient at bundling commands than the HTTP version of AniDB. Simply due to usage patterns and people not having to load a html page.

As for HTTP load, the tcp api which aom uses is severely limited. If too many clients are working against it connections will be delayed or even refused. I think currently at most 3 clients are allowed to connect at the same time. If a 4th client connects it's placed in a queue. The client is nonagressive when placed in a queue for too long and has a delay to prevent hammering.

I have already asked exp for some way to file multiple changes in one command but it has not happened. I'm sorry but I'm already a step ahead of you plus you don't really understand how the tcp api works. The "HTTP POST" command logic doesn't really work.

The only thing aom could do better atm is store up change requests and file them in batches. This is however only really needed when a major change is done in the creq api. The most recent aom version has already done most of the change requests making that problem go away for the moment.
According to the API statistics AoM was twice as heavy as normal on the api for about 24 hours after 0.5.5 release then returned to normal. I don't know about you but I didn't notice any performance change on anidb.

Change requests comprise 0.11% of the commands sent via the TCP API. Most requests are people changing stuff in their mylist, that is batched properly.

Devil Doll
Posts: 49
Joined: Sat Mar 26, 2005 1:29 pm

Post by Devil Doll » Sun Aug 07, 2005 9:30 pm

PetriW wrote:As for HTTP load, the tcp api which aom uses is severely limited.
Please accept my apology for not reading the specification of the TCP API specification. It was not my intention to critizise your fine client unfairly, I just tried to point out where the observed behaviour (such as reporting a connection for each and every file change) caused me to believe there were room for improvements. But if the TCP interface isn't an equivalent of the HTTP interface, my posting didn't have a point. I'd have a number of other comments on the client but don't feel like posting them now.

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Sun Aug 07, 2005 9:52 pm

Devil Doll wrote:
PetriW wrote:As for HTTP load, the tcp api which aom uses is severely limited.
Please accept my apology for not reading the specification of the TCP API specification. It was not my intention to critizise your fine client unfairly, I just tried to point out where the observed behaviour (such as reporting a connection for each and every file change) caused me to believe there were room for improvements. But if the TCP interface isn't an equivalent of the HTTP interface, my posting didn't have a point. I'd have a number of other comments on the client but don't feel like posting them now.
It is well known it creates a bit too many connections while requesting changes but the client also does not normally have so many change requests to submit. It is already down to the usual usage pattern and in a week or two it should be even lower as I've made tweaks to the change request stuff.

Locked