Any way to get rid of some HTTP overhead?

Want to help out? Need help accessing the AniDB API? This is the place to ask questions.

Moderator: AniDB

Locked
rowaasr13
Posts: 415
Joined: Sat Sep 27, 2003 4:57 am

Any way to get rid of some HTTP overhead?

Post by rowaasr13 »

UDP API is limited and TCP API is not open. Besides, both of them are not well-suited for firewalled enviroment, so I just emulate HTTP user agent with help of perl LWP module in my script that helps me automate some CREQs and mylist adds/changes. Question is: is there anything to lower amount of transfered data except nonav=1 and gzip that I already use?

Some of pretty heavy parts include groups list on creq page - I could use ability to replace it with box to enter group ID - that would save ~150k (or ~60k gzipped) for every CREQ everytime I mass update something either manually or automatically.
Rar
AniDB Staff
Posts: 1471
Joined: Fri Mar 12, 2004 2:41 pm
Location: UK
Contact:

Post by Rar »

If you're up for it rows. I'd say the main way would be by continuing the css conversion and moving a lot more stuff off the page and into the template.

Rar
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Post by exp »

you're doing scripted creqs? x_X
anyway, using the HTTP pages via automated scripts isn't recommended.
it can cause use quite some issues if one script is f00bared or too many ppl use it.
however, additional steps in the direction of a full css conversion would help here too i guess :P

BYe!
EXP
rowaasr13
Posts: 415
Joined: Sat Sep 27, 2003 4:57 am

Post by rowaasr13 »

Yes, most of my recent "File" CREQs are automated. :) You may not remember it now, but I once told you that I keep my own simple DB of data on files that I have or had. Since it is too much pain to scan visually and type this data into DB by hand, I've scripted it and now I'm slowly feeding content of my base by series into AniDB. I check those creqs after they submitted though to make sure they're ok. I don't distribute this script and use it personally, so there's no risk of overuse.

As I've mentioned in first post, right now heaviest chunk of data I don't need, but have to DL everytime is long number<->groupname list of <option>s for addf.group, addf.group2 and addf.group3 fields. Right now it is around 150k unpacked for every CREQ and it will only grow over time. If there were some additonal parameter like nonav to replace those with single type="text" box with just group numeric ID, that would help to cut overhead of each CREQ greatly.

Actually, if entire group selection in CREQ/add could be moved to some sort of static, preferably cacheable pop-up, that would return group id to the form, this would help regular CREQs/adds as well.
fahrenheit
AniDB Staff
Posts: 438
Joined: Thu Apr 08, 2004 1:43 am
Location: Portugal

Post by fahrenheit »

i think i may be wrong but anyway:

after logon all the the creqs work is done by a POST, so i think, you can make your own request trough the http protocol if you have all the id's that are needed to do a creq.

example:

http://anidb.info/perl-bin/animedb.pl?s ... doadd=true

this added this file to the test anime.

If your script does things like this it isn't necessary for you to download the grouplist, also you can make your own interface.

but i guess i'm missing the point no?
rowaasr13
Posts: 415
Joined: Sat Sep 27, 2003 4:57 am

Post by rowaasr13 »

Not exactly - this way of course works too, but you have to fill all fields, including group so you still have to get ID somewhere. It could be extracted from group info URL on file info page, of course, but requesting CREQ form and just pulling out all its fields, changing what you need and dumping back is cleaner, more error proof and less site design dependant than parsing file info page and then generating CREQ form yourself.
Locked