some bugs in the UDP API
Moderator: AniDB
some bugs in the UDP API
I have just terminate the first fase of the development of my Anidb UDP client protocol with Twisted and Python (anidblib).
During tests I have found some bugs.
Content-type: text/x-rst
Anidb UDP API BUGS
=============
Basic
------
* Content encoding
- the current character encoding is us-ascii?
- the escape scheme for command option values is not html form encoding. Only "&" needs to be escaped with "&"
- typo: "<br>" should be "<br />"
- "'" (0x27) is silently replaced by "´" (0x60), so it never appears in strings
* General
- the 555 message sometimes use the "wrong" tag
- response data has always a trailing empty line, this should be documented.
* Flood Protection
- the 555 messages is issued too frequently. My client follows all documented requirements, maybe there are additional undocumented restrictions?
Authing Commands
----------------------
* AUTH
- username should be converted to lowercase, but what about password?
* LOGOUT
- typo: "recieve" should be "receive"
Notify Commands
---------------
typo: "from other the" should be "from the other"
* PUSH
- 271 and 272 are not present in the return codes list
- why the field order in 271 and 272 differs from that in 293/292?. Even the names used in the documentation differs!
- why not just send all the available data with the notifications (as the message body)?
- since MESSAGE command limits the length of the subject and the body, maybe this should be documented here too.
* NOTIFYACK
- the 281 return code is present in the return codes list but not documented here.
Data Commands
-------------------
* ANIME
- romaji/kanji/english/other/synonym/short name are globally unique?
* EPISODE
- special character used in epno should be documented
* FILE
- the field {str anidbfilename} is not selectable by fcode; this means that it is not returned when acode or fcode is given
- the file selection by anime, group and epno may identify more than one file, but only one is returned.
Misc Commands
------------------
* STATS
- the returned data contains 7 fields, not 6; moreover there is a field documented(?) as int4 but, actually, of about 46 bits ::
[['3657', '46120', '169049', '3174', '140781', '35972377698960', '168']]
- creqs should be documented
* TOP
- returned fields should be documented
[list][/list][list=][/list]
During tests I have found some bugs.
Content-type: text/x-rst
Anidb UDP API BUGS
=============
Basic
------
* Content encoding
- the current character encoding is us-ascii?
- the escape scheme for command option values is not html form encoding. Only "&" needs to be escaped with "&"
- typo: "<br>" should be "<br />"
- "'" (0x27) is silently replaced by "´" (0x60), so it never appears in strings
* General
- the 555 message sometimes use the "wrong" tag
- response data has always a trailing empty line, this should be documented.
* Flood Protection
- the 555 messages is issued too frequently. My client follows all documented requirements, maybe there are additional undocumented restrictions?
Authing Commands
----------------------
* AUTH
- username should be converted to lowercase, but what about password?
* LOGOUT
- typo: "recieve" should be "receive"
Notify Commands
---------------
typo: "from other the" should be "from the other"
* PUSH
- 271 and 272 are not present in the return codes list
- why the field order in 271 and 272 differs from that in 293/292?. Even the names used in the documentation differs!
- why not just send all the available data with the notifications (as the message body)?
- since MESSAGE command limits the length of the subject and the body, maybe this should be documented here too.
* NOTIFYACK
- the 281 return code is present in the return codes list but not documented here.
Data Commands
-------------------
* ANIME
- romaji/kanji/english/other/synonym/short name are globally unique?
* EPISODE
- special character used in epno should be documented
* FILE
- the field {str anidbfilename} is not selectable by fcode; this means that it is not returned when acode or fcode is given
- the file selection by anime, group and epno may identify more than one file, but only one is returned.
Misc Commands
------------------
* STATS
- the returned data contains 7 fields, not 6; moreover there is a field documented(?) as int4 but, actually, of about 46 bits ::
[['3657', '46120', '169049', '3174', '140781', '35972377698960', '168']]
- creqs should be documented
* TOP
- returned fields should be documented
[list][/list][list=][/list]
-
- AniDB Staff
- Posts: 438
- Joined: Thu Apr 08, 2004 1:43 am
- Location: Portugal
Re: some bugs in the UDP API
* Content encoding
- the current character encoding is us-ascii?
: actualy it can be anything you suport, but by default, it's ascii.
see here
- typo: "<br>" should be "<br />"
: already on the wiki here
* General
- the 555 message sometimes use the "wrong" tag
: the 555 message is a debug message, in real world it should not appear
- response data has always a trailing empty line, this should be documented.
: Lines are terminated by a \n (no dos linefeed \r).
* Flood Protection
- the 555 messages is issued too frequently. My client follows all documented requirements, maybe there are additional undocumented restrictions?
: that i don't know, i allways played safe and waited 4 seconds between packets in the short burst mode, but if you don't log out, you can have problems, so my advice to you, is to try and develop your own server with a couple of random replys and test a bit offline.
Authing Commands
----------------------
* AUTH
- username should be converted to lowercase, but what about password?
: i think, it's converted also to lowercase, at least i convert mines
* LOGOUT
- typo: "recieve" should be "receive"
: there are more typos than that
Notify Commands
---------------
typo: "from other the" should be "from the other"
* PUSH
- why not just send all the available data with the notifications (as the message body)?
: you may want to just check?
- since MESSAGE command limits the length of the subject and the body, maybe this should be documented here too.
: because, what comes from the website, can and probably will exceed those values, the 100 + 1200 limit it's more related to 1500 bytes - headers of the udp packet than any actual limiting, though i've allready talked to epox about that and he said we should try to go by those limits
* NOTIFYACK
- the 281 return code is present in the return codes list but not documented here.
: i think epox merged some return codes since v2
Misc Commands
------------------
* STATS
- the returned data contains 7 fields, not 6; moreover there is a field documented(?) as int4 but, actually, of about 46 bits ::
[['3657', '46120', '169049', '3174', '140781', '35972377698960', '168']]
: the question is that epox sometimes extends the replys, check the udp dev page to see if there's anything there.
* TOP
- returned fields should be documented
: also it's a wiki, you can document it
- the current character encoding is us-ascii?
: actualy it can be anything you suport, but by default, it's ascii.
see here
- typo: "<br>" should be "<br />"
: already on the wiki here
* General
- the 555 message sometimes use the "wrong" tag
: the 555 message is a debug message, in real world it should not appear
- response data has always a trailing empty line, this should be documented.
: Lines are terminated by a \n (no dos linefeed \r).
* Flood Protection
- the 555 messages is issued too frequently. My client follows all documented requirements, maybe there are additional undocumented restrictions?
: that i don't know, i allways played safe and waited 4 seconds between packets in the short burst mode, but if you don't log out, you can have problems, so my advice to you, is to try and develop your own server with a couple of random replys and test a bit offline.
Authing Commands
----------------------
* AUTH
- username should be converted to lowercase, but what about password?
: i think, it's converted also to lowercase, at least i convert mines
* LOGOUT
- typo: "recieve" should be "receive"
: there are more typos than that
Notify Commands
---------------
typo: "from other the" should be "from the other"
* PUSH
- why not just send all the available data with the notifications (as the message body)?
: you may want to just check?
- since MESSAGE command limits the length of the subject and the body, maybe this should be documented here too.
: because, what comes from the website, can and probably will exceed those values, the 100 + 1200 limit it's more related to 1500 bytes - headers of the udp packet than any actual limiting, though i've allready talked to epox about that and he said we should try to go by those limits
* NOTIFYACK
- the 281 return code is present in the return codes list but not documented here.
: i think epox merged some return codes since v2
Misc Commands
------------------
* STATS
- the returned data contains 7 fields, not 6; moreover there is a field documented(?) as int4 but, actually, of about 46 bits ::
[['3657', '46120', '169049', '3174', '140781', '35972377698960', '168']]
: the question is that epox sometimes extends the replys, check the udp dev page to see if there's anything there.
* TOP
- returned fields should be documented
: also it's a wiki, you can document it
Since I use Python, I have no problems at using ascii plus xml charachers entity!fahrenheit wrote:* Content encoding
- the current character encoding is us-ascii?
: actualy it can be anything you suport, but by default, it's ascii.
see here
But in the database what encoding is used?
And, more important, in the client the connect to the database?
Well, what do you mean by "real world"?fahrenheit wrote: * General
- the 555 message sometimes use the "wrong" tag
: the 555 message is a debug message, in real world it should not appear
A client marked as ufficial?
Yes, ok.fahrenheit wrote: - response data has always a trailing empty line, this should be documented.
: Lines are terminated by a \n (no dos linefeed \r).
But *there is* a trailing '\n' at the end of the data and this is not documented.
Now I follow the API reccomendation of 0.5 packets per second.fahrenheit wrote: * Flood Protection
- the 555 messages is issued too frequently. My client follows all documented requirements, maybe there are additional undocumented restrictions?
: that i don't know, i allways played safe and waited 4 seconds between packets in the short burst mode, but if you don't log out, you can have problems, so my advice to you, is to try and develop your own server with a couple of random replys and test a bit offline.
But there is no problem at use a safer frequency when several packats are sent.
Just document it!
Yes, but its just the message body and notification type that are added in NOTIFYGET.fahrenheit wrote: * PUSH
- why not just send all the available data with the notifications (as the message body)?
: you may want to just check?
There is nothing there.fahrenheit wrote: Misc Commands
------------------
* STATS
- the returned data contains 7 fields, not 6; moreover there is a field documented(?) as int4 but, actually, of about 46 bits ::
[['3657', '46120', '169049', '3174', '140781', '35972377698960', '168']]
: the question is that epox sometimes extends the replys, check the udp dev page to see if there's anything there.
But I don't know what they are!fahrenheit wrote: * TOP
- returned fields should be documented
: also it's a wiki, you can document it
I'm not a full anidb user.
I just use the database to check my downloaded files.
Thanks and regards
-
- AniDB Staff
- Posts: 379
- Joined: Sun Nov 07, 2004 11:05 am
the db encoding is ascii, but that's irrelevant. just use whatever with 'enc=' in AUTH (like enc=utf8). this is not included in the wiki yet, though.
the 555 message is just enabled to help developers understand what they are doing wrong. normally the server will just go silent without any notice. and... when you are banned, your opinions and needs obviously don't matter anymore. i mean who cares about your lil tag then
i know the anti leech banning has been aggressive the last week. this will be adjusted. however, there's nothing wrong with the flood-ban. remember to include transfer delays in the calculation. 2 sec on your side doesn't have to be 2 sec on the server side
TOP is documented:
the password is case sensitive, afaik
"- romaji/kanji/english/other/synonym/short name are globally unique?" yes. case sensitive, but aname is not..
"- the field {str anidbfilename} is not selectable by fcode; this means that it is not returned when acode or fcode is given" yes, and that was intended. when you use acode and fcode you should be able to build it yourself.
i'll fix/add the other stuff in the wiki later, unless somebody else does it before me
thanks for the report
the 555 message is just enabled to help developers understand what they are doing wrong. normally the server will just go silent without any notice. and... when you are banned, your opinions and needs obviously don't matter anymore. i mean who cares about your lil tag then
i know the anti leech banning has been aggressive the last week. this will be adjusted. however, there's nothing wrong with the flood-ban. remember to include transfer delays in the calculation. 2 sec on your side doesn't have to be 2 sec on the server side
TOP is documented:
(and the strings are user names)wiki wrote:Possible Replies:
207 TOP
{str longest mylist}|{int count}
{str largest mylist}|{int count}
{str most lame files}|{int count}
{str most indep. user}|{int count}
{str biggest leecher}|{int count}
{str most anime added}|{int count}
{str most eps added}|{int count}
{str most files added}|{int count}
{str most groups added}|{int count}
{str most votes}|{int count}
{str most reviews}|{int count}
the password is case sensitive, afaik
"- romaji/kanji/english/other/synonym/short name are globally unique?" yes. case sensitive, but aname is not..
"- the field {str anidbfilename} is not selectable by fcode; this means that it is not returned when acode or fcode is given" yes, and that was intended. when you use acode and fcode you should be able to build it yourself.
i'll fix/add the other stuff in the wiki later, unless somebody else does it before me
thanks for the report
No problemsepoximator wrote:the 555 message is just enabled to help developers understand what they are doing wrong. normally the server will just go silent without any notice. and... when you are banned, your opinions and needs obviously don't matter anymore. i mean who cares about your lil tag then
All I need to know is:
- is the 0.5 packets per seconds limit safe?
Should I use 1?
As an example, this means I can do 52 requests in 26 seconds?
- what do you intend for "long term"?
The extented amount of time refer to the client uptime?
This means, as an example, that if I got a 271 NOTIFICATION after 1 hour, I have do a PUSHACK, NOTIFYLIST, NOTIFYGET, NOTIFYACK in 2 minutes (each request after 30 seconds)?
Thanks and regards
-
- AniDB Staff
- Posts: 379
- Joined: Sun Nov 07, 2004 11:05 am
u know, 0.5 packets per sec is not possible. it means 1 packet per 2 seconds. that's max 13 request in 26 sec. add 200 ms per req to be sure.
long term: it's really a question about number of request per time. when only checking notifies once in a while, you don't have to worry about this.
but, let's say you want to add 4k files to mylist with FILE and MYLISTADD, then you _should_ use a longer delay. this can, in a way, only be considered as good manners since the server doesn't enforce any long term policy directly. however, there is the anti leech system, that will kick in sooner or later if you abuse the service. there are no official rules when it comes to leeching. use common sense
i'm interested in 6xx errors, or what command you used to provoke them. known issue: non ascii characters in episode names will generate an internal error message. this shouldn't happen too often nowadays, though
long term: it's really a question about number of request per time. when only checking notifies once in a while, you don't have to worry about this.
but, let's say you want to add 4k files to mylist with FILE and MYLISTADD, then you _should_ use a longer delay. this can, in a way, only be considered as good manners since the server doesn't enforce any long term policy directly. however, there is the anti leech system, that will kick in sooner or later if you abuse the service. there are no official rules when it comes to leeching. use common sense
i'm interested in 6xx errors, or what command you used to provoke them. known issue: non ascii characters in episode names will generate an internal error message. this shouldn't happen too often nowadays, though
Ok, sorry!epoximator wrote:u know, 0.5 packets per sec is not possible. it means 1 packet per 2 seconds. that's max 13 request in 26 sec. add 200 ms per req to be sure.
So, this was the reason for getting banned and banned...
Well, in my case I only request FILE and EPISODE for an anime serie once a while;epoximator wrote: long term: it's really a question about number of request per time. when only checking notifies once in a while, you don't have to worry about this.
but, let's say you want to add 4k files to mylist with FILE and MYLISTADD, then you _should_ use a longer delay. this can, in a way, only be considered as good manners since the server doesn't enforce any long term policy directly. however, there is the anti leech system, that will kick in sooner or later if you abuse the service. there are no official rules when it comes to leeching. use common sense
but I would like to add this protection in my protocol implementation (that I will use to write a pair of clients)
This was during test (where I run the same scripts more than one time); the first time I obtain a 600 after an AUTH and STATS commands.epoximator wrote: i'm interested in 6xx errors, or what command you used to provoke them. known issue: non ascii characters in episode names will generate an internal error message. this shouldn't happen too often nowadays, though
Thanks and regards
-
- AniDB Staff
- Posts: 379
- Joined: Sun Nov 07, 2004 11:05 am
Code: Select all
> SENDMSG to=epoximator&title=add&body=5+8=13&s=xxxxx
< 294 SENDMSG SUCCESSFUL