4GB limit? [trackered]

already fixed bugs

Moderator: AniDB

Locked
Isochroma
Posts: 27
Joined: Sat Aug 28, 2004 2:31 am

4GB limit? [trackered]

Post by Isochroma »

I understand that the lack of manpower may be a problem; with thousands of lines of code to look through, it may be a daunting job to fix such a large problem.

Such a big job might be beyond the powers of eXp et al., or maybe not?

Since I have free time and programming skills, how about sending me the code segments that contain faults, and I can check through and fix the errors myself? I would be glad to help the cause and contribute to the excellent system you have built!

If lack of funds is the issue, I can send a specified amount in $USD via postal money order to an address, as well. I am very interested in seeing this problem fixed... What would be the dollar cost if I were to hire you to fix it?

Thanks.
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Post by exp »

well,

the problem is simply the dataformat which is currently used to store the size info.
it's a signed 4 byte integer field.
that limits the filesize to ~ 2^31

unfortunatly switching to unsigned 4 byte fields or 8 byte fields would probably be a real pain. don't get me wrong here, there are probably no code parts in anidb which would need to be changed for this.
however, switching to 8 byte fields would increase the database size and ram usage. I don't know how well perl would handle those either. I don't think that the few files which are larger than 2GB warrant this.

BYe!
EXP
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

Here is the root of the problem: Size limit on files in AniDB / Invalid ID [OPEN]. Basically the system needs to be changed to do one of these EVERYWHERE the files size is used:
1) Change all of the number handling to string/float operations
2) Use a module to handle the larger numbers
3) Get the underlying perl to use 64-bit integers instead of signed 32-bits
an incomplete solution would only cause more problems than it fixes, so this is something that would need to be done carefully.

As far as working on the code, first you need to gain exp's trust and so far you appear to be off to a bad start. As a workaround currently I suggest you use the generic file to add the episode to your mylist for the time being.

[ :? I got interrupted and took a while to type this up and exp ended up responding first...]
Isochroma
Posts: 27
Joined: Sat Aug 28, 2004 2:31 am

Possibilities

Post by Isochroma »

Sounds a bit more complex. I am sure that in the near future, many files will be posted that are larger than 2GB. With the coming HD-DVD, Blu-Ray Disc, and other storage technologies such as HVD (Holographic Versatile Disc), files will only be increasing in size.

You could test the system and check to see what RAM/disk usage would be... size fields couldn't take up that much space, could they? Consider though that as time passes, it will only get more difficult to make the change, so why not switch now?

If you can implement 8-byte size fields or whatever size is required to fully support the ED2K maximum file size of 4,290,048,000 bytes in the AniDB by 31/03/2005, I will send a postal money order or wire the funds directly to an account of your choosing in the amount of $100 USD.

I am also open to negotiation on the cost of your time to do this job. My funding is flexible.

Looking forward to hearing your reply, and thanks for your consideration and time.
kidan
Posts: 319
Joined: Thu Feb 13, 2003 9:13 pm
Location: .DE

Post by kidan »

Well, as siad before AniDB is mainly fansub-centered and fansubbers tend to use codecs which give best quality-to-filesize-ratio. currently a normal fansub ep. ia about 170 MB to 250 MB. Dvdrips are usually encoded to xvids with 700 MB or 2x700 MB. Even with HD-DVD or other bigger storage-media this would not increase dramatically, as to some extend the eye cannot see the difference when watched on a screen/TV/beamer, as those limit the resolution. And as FS-groups usually keep an eye on filesize/bandwidth, I guess filesizes might triple some day, but not that much more in near future.
Remember: We do not catalogue 1 to 1 DVD-rips here.
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Re: Possibilities

Post by exp »

Well,

right now switching to 8 byte integers is not ONTODO, however, should such large files really start to appear in numbers I'll rethink about making this change.
however, currently I don't think it's worth the extra strain on the server and the time it needs to implement & test.
Isochroma wrote:If you can implement 8-byte size fields or whatever size is required to fully support the ED2K maximum file size of 4,290,048,000 bytes in the AniDB by 31/03/2005, I will send a postal money order or wire the funds directly to an account of your choosing in the amount of $100 USD.
sorry, I am doing this in my free time out of personal interest and I don't plan to change that.
however as anidb is constantly growing maybe we'll need to start gathering donations some day to handle the hosting costs.
I'd then certainly appreciate if you donate :o)
However, donations would not affect the priority of feature requests. :P

BYe!
EXP
epoximator
AniDB Staff
Posts: 379
Joined: Sun Nov 07, 2004 11:05 am

Post by epoximator »

where's my money! ah, two years too late
Isochroma
Posts: 27
Joined: Sat Aug 28, 2004 2:31 am

Post by Isochroma »

A bit broke right now, due to unmentionable circumstances :(

It's great to see the coders have found time to make this change, thanks! It will be very useful for so many files now & in the future :)

I guess this thread can be closed now too. Bye & keep up the good work!
Locked