some bugs in the UDP API

Want to help out? Need help accessing the AniDB API? This is the place to ask questions.

Moderator: AniDB

Locked
darsh
Posts: 12
Joined: Wed Apr 12, 2006 6:46 pm

some bugs in the UDP API

Post by darsh »

I have just terminate the first fase of the development of my Anidb UDP client protocol with Twisted and Python (anidblib).

During tests I have found some bugs.


Content-type: text/x-rst

Anidb UDP API BUGS
=============

Basic
------

* Content encoding

- the current character encoding is us-ascii?

- the escape scheme for command option values is not html form encoding. Only "&" needs to be escaped with "&"

- typo: "<br>" should be "<br />"

- "'" (0x27) is silently replaced by "´" (0x60), so it never appears in strings

* General

- the 555 message sometimes use the "wrong" tag

- response data has always a trailing empty line, this should be documented.

* Flood Protection

- the 555 messages is issued too frequently. My client follows all documented requirements, maybe there are additional undocumented restrictions?


Authing Commands
----------------------

* AUTH

- username should be converted to lowercase, but what about password?

* LOGOUT

- typo: "recieve" should be "receive"


Notify Commands
---------------

typo: "from other the" should be "from the other"

* PUSH

- 271 and 272 are not present in the return codes list

- why the field order in 271 and 272 differs from that in 293/292?. Even the names used in the documentation differs!

- why not just send all the available data with the notifications (as the message body)?

- since MESSAGE command limits the length of the subject and the body, maybe this should be documented here too.

* NOTIFYACK

- the 281 return code is present in the return codes list but not documented here.


Data Commands
-------------------

* ANIME

- romaji/kanji/english/other/synonym/short name are globally unique?

* EPISODE

- special character used in epno should be documented

* FILE

- the field {str anidbfilename} is not selectable by fcode; this means that it is not returned when acode or fcode is given

- the file selection by anime, group and epno may identify more than one file, but only one is returned.


Misc Commands
------------------

* STATS

- the returned data contains 7 fields, not 6; moreover there is a field documented(?) as int4 but, actually, of about 46 bits ::

[['3657', '46120', '169049', '3174', '140781', '35972377698960', '168']]

- creqs should be documented

* TOP

- returned fields should be documented





































[list][/list][list=][/list]
fahrenheit
AniDB Staff
Posts: 438
Joined: Thu Apr 08, 2004 1:43 am
Location: Portugal

Re: some bugs in the UDP API

Post by fahrenheit »

* Content encoding

- the current character encoding is us-ascii?

: actualy it can be anything you suport, but by default, it's ascii.
see here

- typo: "<br>" should be "<br />"

: already on the wiki here

* General

- the 555 message sometimes use the "wrong" tag

: the 555 message is a debug message, in real world it should not appear

- response data has always a trailing empty line, this should be documented.

: Lines are terminated by a \n (no dos linefeed \r).

* Flood Protection

- the 555 messages is issued too frequently. My client follows all documented requirements, maybe there are additional undocumented restrictions?

: that i don't know, i allways played safe and waited 4 seconds between packets in the short burst mode, but if you don't log out, you can have problems, so my advice to you, is to try and develop your own server with a couple of random replys and test a bit offline.

Authing Commands
----------------------

* AUTH

- username should be converted to lowercase, but what about password?

: i think, it's converted also to lowercase, at least i convert mines

* LOGOUT

- typo: "recieve" should be "receive"

: there are more typos than that :P

Notify Commands
---------------

typo: "from other the" should be "from the other"

* PUSH

- why not just send all the available data with the notifications (as the message body)?

: you may want to just check?

- since MESSAGE command limits the length of the subject and the body, maybe this should be documented here too.

: because, what comes from the website, can and probably will exceed those values, the 100 + 1200 limit it's more related to 1500 bytes - headers of the udp packet than any actual limiting, though i've allready talked to epox about that and he said we should try to go by those limits

* NOTIFYACK

- the 281 return code is present in the return codes list but not documented here.

: i think epox merged some return codes since v2

Misc Commands
------------------

* STATS

- the returned data contains 7 fields, not 6; moreover there is a field documented(?) as int4 but, actually, of about 46 bits ::

[['3657', '46120', '169049', '3174', '140781', '35972377698960', '168']]

: the question is that epox sometimes extends the replys, check the udp dev page to see if there's anything there.

* TOP

- returned fields should be documented

: also it's a wiki, you can document it :P
darsh
Posts: 12
Joined: Wed Apr 12, 2006 6:46 pm

Post by darsh »

fahrenheit wrote:* Content encoding

- the current character encoding is us-ascii?

: actualy it can be anything you suport, but by default, it's ascii.
see here
Since I use Python, I have no problems at using ascii plus xml charachers entity!

But in the database what encoding is used?
And, more important, in the client the connect to the database?

fahrenheit wrote: * General

- the 555 message sometimes use the "wrong" tag

: the 555 message is a debug message, in real world it should not appear
Well, what do you mean by "real world"?
A client marked as ufficial?

fahrenheit wrote: - response data has always a trailing empty line, this should be documented.

: Lines are terminated by a \n (no dos linefeed \r).
Yes, ok.
But *there is* a trailing '\n' at the end of the data and this is not documented.

fahrenheit wrote: * Flood Protection

- the 555 messages is issued too frequently. My client follows all documented requirements, maybe there are additional undocumented restrictions?

: that i don't know, i allways played safe and waited 4 seconds between packets in the short burst mode, but if you don't log out, you can have problems, so my advice to you, is to try and develop your own server with a couple of random replys and test a bit offline.
Now I follow the API reccomendation of 0.5 packets per second.
But there is no problem at use a safer frequency when several packats are sent.
Just document it!

fahrenheit wrote: * PUSH

- why not just send all the available data with the notifications (as the message body)?

: you may want to just check?
Yes, but its just the message body and notification type that are added in NOTIFYGET.
fahrenheit wrote: Misc Commands
------------------

* STATS

- the returned data contains 7 fields, not 6; moreover there is a field documented(?) as int4 but, actually, of about 46 bits ::

[['3657', '46120', '169049', '3174', '140781', '35972377698960', '168']]

: the question is that epox sometimes extends the replys, check the udp dev page to see if there's anything there.
There is nothing there.
fahrenheit wrote: * TOP

- returned fields should be documented

: also it's a wiki, you can document it :P
But I don't know what they are!
I'm not a full anidb user.
I just use the database to check my downloaded files.



Thanks and regards
epoximator
AniDB Staff
Posts: 379
Joined: Sun Nov 07, 2004 11:05 am

Post by epoximator »

the db encoding is ascii, but that's irrelevant. just use whatever with 'enc=' in AUTH (like enc=utf8). this is not included in the wiki yet, though.

the 555 message is just enabled to help developers understand what they are doing wrong. normally the server will just go silent without any notice. and... when you are banned, your opinions and needs obviously don't matter anymore. i mean who cares about your lil tag then

i know the anti leech banning has been aggressive the last week. this will be adjusted. however, there's nothing wrong with the flood-ban. remember to include transfer delays in the calculation. 2 sec on your side doesn't have to be 2 sec on the server side

TOP is documented:
wiki wrote:Possible Replies:
207 TOP
{str longest mylist}|{int count}
{str largest mylist}|{int count}
{str most lame files}|{int count}
{str most indep. user}|{int count}
{str biggest leecher}|{int count}
{str most anime added}|{int count}
{str most eps added}|{int count}
{str most files added}|{int count}
{str most groups added}|{int count}
{str most votes}|{int count}
{str most reviews}|{int count}
(and the strings are user names)

the password is case sensitive, afaik

"- romaji/kanji/english/other/synonym/short name are globally unique?" yes. case sensitive, but aname is not..

"- the field {str anidbfilename} is not selectable by fcode; this means that it is not returned when acode or fcode is given" yes, and that was intended. when you use acode and fcode you should be able to build it yourself.

i'll fix/add the other stuff in the wiki later, unless somebody else does it before me

thanks for the report
pelican
AniDB Staff
Posts: 234
Joined: Wed Aug 11, 2004 11:19 pm

Post by pelican »

Query volume might perhaps be easier to manage if you added a warning message which clients could respond to with a particular delay after which they're guaranteed to be safe.
darsh
Posts: 12
Joined: Wed Apr 12, 2006 6:46 pm

Post by darsh »

epoximator wrote:the 555 message is just enabled to help developers understand what they are doing wrong. normally the server will just go silent without any notice. and... when you are banned, your opinions and needs obviously don't matter anymore. i mean who cares about your lil tag then
No problems ;-)

All I need to know is:
- is the 0.5 packets per seconds limit safe?
Should I use 1?
As an example, this means I can do 52 requests in 26 seconds?

- what do you intend for "long term"?
The extented amount of time refer to the client uptime?
This means, as an example, that if I got a 271 NOTIFICATION after 1 hour, I have do a PUSHACK, NOTIFYLIST, NOTIFYGET, NOTIFYACK in 2 minutes (each request after 30 seconds)?


Thanks and regards
darsh
Posts: 12
Joined: Wed Apr 12, 2006 6:46 pm

Post by darsh »

I just forget: for two times I got a 600 error.


regards[/b][/i]
epoximator
AniDB Staff
Posts: 379
Joined: Sun Nov 07, 2004 11:05 am

Post by epoximator »

u know, 0.5 packets per sec is not possible. it means 1 packet per 2 seconds. that's max 13 request in 26 sec. add 200 ms per req to be sure.

long term: it's really a question about number of request per time. when only checking notifies once in a while, you don't have to worry about this.

but, let's say you want to add 4k files to mylist with FILE and MYLISTADD, then you _should_ use a longer delay. this can, in a way, only be considered as good manners since the server doesn't enforce any long term policy directly. however, there is the anti leech system, that will kick in sooner or later if you abuse the service. there are no official rules when it comes to leeching. use common sense

i'm interested in 6xx errors, or what command you used to provoke them. known issue: non ascii characters in episode names will generate an internal error message. this shouldn't happen too often nowadays, though
darsh
Posts: 12
Joined: Wed Apr 12, 2006 6:46 pm

Post by darsh »

epoximator wrote:u know, 0.5 packets per sec is not possible. it means 1 packet per 2 seconds. that's max 13 request in 26 sec. add 200 ms per req to be sure.
Ok, sorry!
So, this was the reason for getting banned and banned...
epoximator wrote: long term: it's really a question about number of request per time. when only checking notifies once in a while, you don't have to worry about this.

but, let's say you want to add 4k files to mylist with FILE and MYLISTADD, then you _should_ use a longer delay. this can, in a way, only be considered as good manners since the server doesn't enforce any long term policy directly. however, there is the anti leech system, that will kick in sooner or later if you abuse the service. there are no official rules when it comes to leeching. use common sense
Well, in my case I only request FILE and EPISODE for an anime serie once a while;
but I would like to add this protection in my protocol implementation (that I will use to write a pair of clients)
epoximator wrote: i'm interested in 6xx errors, or what command you used to provoke them. known issue: non ascii characters in episode names will generate an internal error message. this shouldn't happen too often nowadays, though
This was during test (where I run the same scripts more than one time); the first time I obtain a 600 after an AUTH and STATS commands.



Thanks and regards
darsh
Posts: 12
Joined: Wed Apr 12, 2006 6:46 pm

Post by darsh »

Sorry, but there seems to be still problems with parameters encoding.

Now the '=' cannot be present (as an example in message body).
I've tried to escape with xml character entity but still does not works (I got a 505 ILLEGAL INPUT OR ACCESS DENIED)


Thanks and regards
epoximator
AniDB Staff
Posts: 379
Joined: Sun Nov 07, 2004 11:05 am

Post by epoximator »

Code: Select all

> SENDMSG to=epoximator&title=add&body=5+8=13&s=xxxxx
< 294 SENDMSG SUCCESSFUL
?
darsh
Posts: 12
Joined: Wed Apr 12, 2006 6:46 pm

Post by darsh »

You are right, sorry.

The problem is not with '=', but with newline:

SENDMSG body=test<br />&to=myself&s=xxx&title=newline


Thanks and regards
epoximator
AniDB Staff
Posts: 379
Joined: Sun Nov 07, 2004 11:05 am

Post by epoximator »

fixed, btw
Locked