Shadow Files Progress [ONTODO/POSTPONED]

old granted and denied feature requests

Moderator: AniDB

Rar
AniDB Staff
Posts: 1471
Joined: Fri Mar 12, 2004 2:41 pm
Location: UK
Contact:

Shadow Files Progress [ONTODO/POSTPONED]

Post by Rar »

An ONTODO request (from Aug 2003!):
http://www.anidb.net/forum/viewtopic.php?t=621

I understand this is a big task and is going to require a pretty major reorganizastion of how files and relations are dealt with.

I'm far from a database expert, but thought a bit of thinking through might help exp along the way.

At the moment AniDB is set up like this right:?
Anime-<Episode-<File
Would changing the Ep-File relationship to >-< both ways help things? Or does having half episode files confuse things?

Rar
PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW »

Having EP to FILE relations doesn't help at all imho, it would just create redundant data as files already know what EP they belong to.
What's needed is essentially an expanded FILE to EP relation where files can both be flagged as part of a file (aka part 2/3) and belong to multiple episodes. There also needs to be an additional FILE to FILE relation for subtitles and the like.
It's pretty easy to code, it's just a significant change and making it user friendly might also be a problem. The big problem is that while it's a simple change so much has been made already and I imagine it would be difficult to implement in several stages.
Skywalka
Posts: 889
Joined: Tue Sep 16, 2003 7:57 pm

Post by Skywalka »

Hmmm...

I do not quite understand how PostreSQL works but wouldn't it be possible to export several Anime to which the shadow file problem applies to a test-AniDB and play around with it there untill it is pretty much working and then make a backup and apply it to AniDB?

I know I am again talking about work others would have to do before I could even touch the test-DB but as always I can only offer my help as a tester so if anything like this is possible I'd try it out molto. ^_^
PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW »

I'm certain EXP has some test thing somewhere but like, I don't think that's the problem either. ;)
The thing is when this is applied all files will be changed, all lists that use files will be changed and the mylist might need to be changed.

The mods will have a fun time editing entries too. 8O
Skywalka
Posts: 889
Joined: Tue Sep 16, 2003 7:57 pm

Post by Skywalka »

yeah sure but the idea of shadow files was first mentioned eight months ago and in the meantime I filed ~ 3800 creqs. If shadow files had been around before that, all the work that was done in the meantime would not have to be changed today. Or let me put it this way - this problem is one of the underlying things - stuff that will have to be fixed one day, or otherwise people won't be able to complete their animes ever - I still have 12/48 Mini Goddesses episodes though those 12 files contain all episodes.

Most of the stuff I requested in the last few weeks tended to be directed towards the impression that deprecated files are not needed and that AniDB should be cleaned from those files - since this is not the case, and I agree to it now, I have to say that the longer we wait for this and the more "deprecated" files are added to AniDB the more work it will be later. Look at the user count, we now have almost tripled the number from last september when I joined and though I am not certain I'd say that the number of files in the DB might have doubled, with the amount of correct files because of CREQs might have increased significant as well.

If it is a problem to handle the CREQs I guess that effect could be taken care of if the admins agree on a certain date when the change happens and they might be around to be able to handle most of those requests that can be expected.

I just get the feeling that the longer this is not tended to the large a problem it will be later this year. I mean look at the number of people who download Naturdo each time a new EP is released. 11000+ users on BT every time now, back in september that number was around 7000. The number of users can only rise and the number of files available with it - I think.

I don't mean to push anyone to do this, but I guess it is a good idea to push a little now :) Just a little.
Iceman[grrrr]
Posts: 312
Joined: Sat Aug 02, 2003 3:22 am
Location: Québec, Canada

Post by Iceman[grrrr] »

Just a question, why didn't you continue the past thread instead of posting a new one ?
Rar
AniDB Staff
Posts: 1471
Joined: Fri Mar 12, 2004 2:41 pm
Location: UK
Contact:

Post by Rar »

@Iceman: Seeing as the last post in the old thread was on Sun Aug 10, I thought a refresher and maybe some new ideas would be helpful.

@PetriW: So are you talking about each episode having 'container' files, which would then have whatever files from that release? But how would you solve multiple episodes in a single file?

Rar
PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW »

Essentially:
EP TABLE <- RELATION TABLE -> FILE TABLE
Atm it looks like this:
EP TABLE <- FILE TABLE
kidan
Posts: 319
Joined: Thu Feb 13, 2003 9:13 pm
Location: .DE

Post by kidan »

If I remember correctly the whole shadow-files-idea was brought up to avoid any change to the database (which would be the only clean solution IMHO). Thus the relations PetriW posted would be nice, but are not the intention of the shadow-files. Shadow-files would integrate into the currently existing DB and would need some special handling in midleware-layer.
PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW »

I'll be honest see no real difference between the two except the one exp listed will be "fun" to implement. ;)

I could explain all about how each relation would have to be made and what columns would be needed in the FILE and EP tables but since that's really not useful information in this case I simplified it. The way I think I place relations in a sepatare table since that's clearer, exp just adds more fields to the existing table (file part, file symlinks, episode folders) and well, in the end exp still needs to implement this and we're still waiting for him to do so.

The question was if there was a use for having EP -> FILE relations.
PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW »

Latest status update:
[2004-03-29 21:43:50] <PetriW> I'm gonna be a pain in the ass
[2004-03-29 21:44:20] <PetriW> and ask
[2004-03-29 21:44:25] <PetriW> can you fix this soon plz?
[2004-03-29 21:44:25] <PetriW> http://www.anidb.net/forum/viewtopic.php?t=621
[2004-03-29 21:44:31] <PetriW> it's like, 2 years overdue
[2004-03-29 21:44:35] <PetriW> ^^
[2004-03-29 21:46:25] * PetriW bets 5 bucks EXP[BUSY] put him on ignore~
[2004-03-29 21:52:46] <EXP[BUSY]> not yet, but i might @ ignore :P
Elberet
Posts: 778
Joined: Sat Jul 19, 2003 8:14 pm

Post by Elberet »

Hmm...
<rambling>
variable level hierarchies and relational databases are worlds apart, so implementing this correctly isn't that easy at all. I'd rather implement it like this:

Code: Select all

ep_table <-- item_table <-+-  -  -  -  -+-- files_table
                          |             |
                          |             +--------+
                          |                      |
                          +-- container_table <--+
(edit by wahaha: fixed the tabs inside

Code: Select all

)[/color][/size]
Items are basically your n:n relation, with the exception that when an episode consists of multiple files, the files are associated to a container and the container is related to the episode instead of multiple files. Otherwhise it might not be possible to properly group files within one episode, e.g. by fansub-group, version, language, ..., or the tests needed to perform this grouping would be very expensive (more expensive then joining over another two- or three-column-table with indices).
</rambling>
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

Added to the tracker.
DonGato
Posts: 1296
Joined: Sun Nov 17, 2002 9:08 pm
Location: The Pampas, The land of the Gaucho!
Contact:

Post by DonGato »

Wasn't this "workarounded" by generics?! O_o
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

DonGato wrote:Wasn't this "workarounded" by generics?! O_o
Yes, generics was a work around. Currently a file is linked to one and exactly one episode. Ideally a single file should be able to be linked to multiple episodes AND multiple files should be able to be linked as different parts of the same episode. The request on the tracker the the latter part.
Locked