Use NTFS ADS (for 0.6)

AniDB Client Feature Requests go here

Moderators: AniDB, AniDB API

Elberet
Posts: 778
Joined: Sat Jul 19, 2003 8:14 pm

Use NTFS ADS (for 0.6)

Post by Elberet » Fri Nov 12, 2004 11:18 pm

ADS are Additional Data Streams. They are a feauture of the NTFS file system on Windows XP (not sure about 2000) and later. These streams are stored alongside of the file and are persistent during the file's life, even if copied or moved to another NTFS volume. Regular file access, e.g. sharing it with a P2P program or watching it, do not interfere with ADS at all.

Now for the actual suggestion...

If a file known to AOM is renamed or moved to a different location (e.g. whenever I move a series to a different folder as soon as it has been completed), AOM needs to re-hash the file again and it will think that it's now a duplicate. The hashing is a fairly expensive operation, however. Hence my suggestion: After hashing a file, store the ed2k-hash and filesize in an ADS for the recently hashed file. After discovering a new file, check if an ADS with an ed2k-hash and filesize exists for the file and, if present, use that hash to identify the file (if the filesize matches). Possibly, also store the file's location where it was first hashed, and if present, check the known-files list for that file, check it's existence, and if not present, assume that the file has been moved and update the known-files list accordingly.

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Fri Nov 12, 2004 11:35 pm

Ooo, I'll look into this asap. Thanks for enlightning me! :D

Elberet
Posts: 778
Joined: Sat Jul 19, 2003 8:14 pm

Post by Elberet » Sat Nov 13, 2004 1:27 am

You're welcome... :)

User avatar
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Post by exp » Sat Nov 13, 2004 8:07 am

this might be usefull, however there should be an option to disable it and a large onetime message popup to the user when the feature is first used.
bc files with additional datastreams will produce weird warning popups when one tries to move them to a non ntfs filesystem.

BYe!
EXP

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Sat Nov 13, 2004 10:51 am

Yeah, it'll be in the first time startup instructions.

I'll probably store size, ed2k, crc32, md5 and sha1 in an xml file in the data stream (for others to read) along with an aom checksum of the values to prevent fiddling.

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Sat Nov 13, 2004 3:43 pm

Oh this owns!
http://msdn.microsoft.com/library/defau ... /ntfs5.asp

Works in windows 2000 too!

egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg » Sat Nov 13, 2004 4:28 pm

PetriW wrote:Oh this owns!
http://msdn.microsoft.com/library/defau ... /ntfs5.asp

Works in windows 2000 too!
I don't have time to read it atm, but you might want to consider adding more information for native/3rd party tools to search for details in these fields... That way even if they are renamed they could still be found based on Anime name, ....

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Sat Nov 13, 2004 5:54 pm

egg wrote:
PetriW wrote:Oh this owns!
http://msdn.microsoft.com/library/defau ... /ntfs5.asp

Works in windows 2000 too!
I don't have time to read it atm, but you might want to consider adding more information for native/3rd party tools to search for details in these fields... That way even if they are renamed they could still be found based on Anime name, ....
Could be a good idea, the space used would be insignificant.

Elberet
Posts: 778
Joined: Sat Jul 19, 2003 8:14 pm

Post by Elberet » Mon Nov 15, 2004 1:39 am

Well, if you store that information in the right way, Windows Explorer itself can make use of it since the author/topic/title information it lets you enter is stored in an ADS as well. :)

Rogelio-60626
Posts: 1
Joined: Sun Aug 21, 2005 11:42 am
Contact:

Post by Rogelio-60626 » Tue Aug 30, 2005 5:49 am

Oh, My God!

e-Viper
Posts: 79
Joined: Mon Jun 07, 2004 12:38 pm
Location: Belgium

Post by e-Viper » Tue Aug 30, 2005 12:38 pm

sounds promicing.

I would have been already happy with another color if the filename wasn't found at the location it was hashed

Amour
Posts: 640
Joined: Fri Oct 08, 2004 5:19 pm
Location: France

Post by Amour » Tue Aug 30, 2005 3:03 pm

I want it.

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Tue Aug 30, 2005 3:41 pm

It's in the beta already.

Amour
Posts: 640
Joined: Fri Oct 08, 2004 5:19 pm
Location: France

Post by Amour » Tue Aug 30, 2005 7:46 pm

By the way, WinFS Beta is available:
http://www.vnunet.com/2141599

And where can I find AOM Beta?

PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW » Tue Aug 30, 2005 9:38 pm

Sadly I don't feel like paying a gazillion for a MSDN subscription, ADS is enough.

Locked