exp wrote:in most cases your client will probably download one chunk from multiple locations and would therefore be able to detect the corrupted chunk.
Indeed... But I wonder if the hybrid's new "chunk"-size (<600KB) also introduced anything but MD4... 600K from a single uploader is probably quite common.
Elberet wrote:Good point(s). However, I wonder whether this is really a problem that's specific to MD4. It doesn't sound like a big difference, time-wise assuming that the "attacker" relies on a brute-force method, whether one creates a chunk of data that matches a given MD4, MD5 or SHA(1) hash.
Exactly that's why MD4 is "broken": It doesn't take the usual brute-force to create a collision with MD4, it can be done in less than a minute or even a few seconds...
Just for information: as far as I have seen, overpeer does not send out bad chunks (still), but bad hashsets (so one or more of the partitial chunk-MD4 are wrong, but all of them still match the complete ed2k-MD4, so the hashset seems valid). This way they can supply a lot of people with wrong hashes for chunks which leads to correct data to be identifiead as corrupted. It is a quite clever attack on the ed2k-protocol, as it only uses little bandwidth and only very few calculations. Those bad clients cycle their ques very frequent, so that very many people who just started a download get to download from them (just a few kb random junk and the faked hashset). So I guess ed2k should better try to use tiger-tree with a more secure hash-algo in its next protocol-versions.
PS: a current ipfilter.dat file and deleting the old hashset really helps a lot when encountering corruption of the same block over and over.
i think adding an extra hash field for SHA-1 hashes would not be a problem.
i will do that.
any objections? or any other hashes which we should also support?
sha1 hashes can now be added for files.
they have to be in a 40char hex encoding for now though.
i'll implement 32char base64 encoding some day, if there is a need for it.
dunno yet.
Oh /me must have been blind (well perhaps it was not there back then, when I added new files to the DB) or my brain played a trick on me as I thought there were only a MD4 and a CRC32 field.
Tigertree gets used in gnutella2 (and I guess, it may be used in future file-sharing networks, as it is a far better concept, than the chunks-system in ed2k. And it is very fast as it uses the tiger algo. (http://www.cs.technion.ac.il/~biham/Reports/Tiger/)
Please feel free to tell me which of these will _never_ be added in the future so I can save some hashing time. All the others I will run on my files from now on since it takes ages to get the files on HDD later from DVD/CD to calculate new hashes every time a new hash entry is added to the DataBase.
I think adding extra hashes such as SHA-1 would be great.
You can already get many Hashes from www.shareaza.com it can download from Gnutella, G2, eDonkey and Bittorrent networks.. It is even capable of swarming all of the netoworks at once if the hash is available.
Bittorrent Hashes have a problem because they include the file name in the hash so if anyone renames the torrent being shared it nolonger matches the hash.
SHA-1 would probably be the best one to add at this point.
ed2k hashes are calculated by hashing every 9500KB chunk of the file, then concatenating the values of those hashes and hashing that concatenation. The upshot of this is: if the partial hashes match, then the file ID will also match. It would be trivial for someone reasonably knowledgable to create a Corrupt-O-Matic peer, if they were evil. This peer would respond to requests for any partial hash for any file, generate a corrupt 9500KB chunk with an identical hash, and serve it up. This would not be detectable in any way by the clients, the only way you'll find out about it is when you play your newly downloaded file and it's full of random static in places (or crashes your player). If you downloaded part of a chunk from an "evil" peer and another part from a genuine source, then the hash would not match, but any complete chunks downloaded off an evil peer would be undetectably corrupted.
If the evil peer did this for any file request, not just specific files, then it would affect everything; the latest warez, music, anime, whatever, anyone that ever downloaded a chunk off this evil peer would end up with a corrupted download (but with matching hashes). Multiply this by thousand evil peers running on multiple machines / IP addresses; and basically, anyone motivated enough to perform an attack like this could completely disable the Overnet network, and most of eDonkey (they would have trouble getting into more "private" servers, so those might survive).
The reason why this is so damaging is the ease with which fake chunks with the same MD4 hash can be generated; this could not be done with MD5, SHA1, etc. I guess everyone will ignore this until someone finally mounts an attack on the network(s) of this nature...
If I remember correctly there were / are some attacks like this going on from the RIAA (Overpeer). Therefore most clients use the ipfilter.dat to deny any connection to/from those attackers.
But for the serverless kademlia-implementation of emule they should really use a more secure algorithm such as tigertree (smaller chunks, faster, more secure).
