AniDB DOWN - NOW UP AGAIN!

All your questions about AniDB belong in here.
No download support!

Moderator: AniDB

taltamir
Posts: 36
Joined: Wed Mar 23, 2005 4:53 am

Post by taltamir »

well, MilesMi really nailed everything. One question though, how much ram do you have? With sufficient ram its better to completely eliminate swap, or set an autoloading ramdrive and put the swap on it for badly written applications (applications that use swap instead of ram).



Just to clarify, Swap in itself is a really bad idea. A good program just access' its files directly, and caches things to ram for increased speed. Swap is using the HDD as ram AS A FALLBACK when you are OUT OF RAM to prevent the computer from crushing from a program trying to claim more ram then there is available (again, badly written program). Ideally though, if data needs to be written to the HDD its better to write it directly rather then writing it as swap.

Proper sequence is:
1. HDD to RAM for caching
2. RAM to CPU for processing backto RAM.
3. repeat 2 while processing a certain peice of data. (well, L2 cache etc are all just ram blocks found at closer distances to the CPU)
4. Use completed calculations, or forward them to a dedicated hardware peice (audio/video) or save to HDD

However, if you run out of ram this sequence will look like this:
1. HDD to RAM for caching, not enough RAM, some cached back to swap (data1) on HDD
2. RAM to CPU for processing of some data (data2).
3. RAM to Swap (data2) to free more RAM.
4. Swap (data1) to RAM
5. RAM to CPU for processing of some data (data1).
6. repeat 2 through 5 until calculation complete.
7. Use completed calculations, or forward them to a dedicated hardware peice (audio/video) or save to HDD

This is obviously meant to be only done in emergency since you are copying from one point on the harddrive to another and then processing that. A practical example would be to get 7z compression tool and try to compress with a dictionary thats too large. If its borderline, you will not the process continuing until you run out of ram, then it starts using swap and suddenly your harddrive starts making noise like crazy while 7z seems to be "stuck". However, if you are trying to compresss a REALLY small file (a few meg) and have a powerful computer, and leave it on for a LOOOONG time you will notice it slowly moving (it might go 1% per hour of operation or some such).

Just to clarify the speed differences. HDD's access time is measures in miliseconds (1/10^3) while ram access speed in 1/10^9. And most HDD have an access time of about 100ms (thats 1/10 of a second). This is a VERY long time for random access, harddrives are much better as SEQUENTIAL transfers (ie, they might be able to transfer a file at 100mb/s)

An example of a badly written program, "spellforce: the order of dawn" needs about 2GB of swap space to avoid random crushes, REGARDLESS of how much ram you have (It doesn't use that much swap space, but it will sometimes try to write to a swap address thats very high, and if you dont have sufficient swap space it will fail to write it because it does not exist). Oh and it uses all that swap space while NEVER using more then 300MB of ram (tested on a 512MB and a 1GB of ram machine).


A situation where using swap will prvevent a crash and will NOT cause your computer to stop responding is if for example you have 500MB of ram, and your OS's user interface takes 100MB, and you run a program that takes 450MB of ram, that program will load into ram forcing parts of the OS into swap, those parts will stop responding (where windows aren't moving, mouseclicks aren't responded to, you know the drill) while the program's thread is being processed. Once said thread is done it will terminate and the OS portions is swap move back into the ram and come back to proper speed.
If said program is NOT stuck in a loop but takes a really long time to finish your computer might look frozen, but will "unfreeze" if you wait a couple hours.
If the program requires parts of the OS that were pushed to swap to complete its operations then it could literaly push the completion time to thosands of years, in which case it's completely frozen, not to mention it will kill your HDD before it finishes.
In fact, most often it is NOT an infinate loop that causes a "freeze" but rather misuse of swap, which windows handles in a very stupid manner.



As for open sourcing...
While open sourcing something allows people to more easily develop exploits, it also allows more people to CATCH exploits. Thats why open source browsers (mozilla, etc) are SAFER then closed source ones (internet explorer) despite the funding difference. Yes if anidb is opened sourced people could more easily develop exploits, specifically, they could try to abuse the autoreport feature of AOM to mess up the database. So keep PARTS of the AOM closed source, specifically, the parts regarding authentification of the AOM client and autoaccepting the change requests made by it would be wise (although it will still be possible to crack them, but a good hacker could do it right now!). By having all the LOYAL members of anidb with skills in cracking and security and programming help anidb will actually become more secure then ever, while at the same time have many more features and have alot of bugs expunged. It has been proven to work time and time again with various open source projects!
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Post by exp »

taltamir wrote:well, MilesMi really nailed everything. One question though, how much ram do you have? With sufficient ram its better to completely eliminate swap, or set an autoloading ramdrive and put the swap on it for badly written applications (applications that use swap instead of ram).
... very lengthy explanation follows ...
Seriously dude, don't transfer your experiences with M$ Win$uck to other OSs. Linux can actually make great use of swap and having it will actually _improve_ your overall system performance.
please note: having swap != trashing

I am administrating a lot of servers and even if it is a 4GB RAM box it can still end up putting a couple of hundred MBs in swap and use that ram for caching of IO requests.
It's perfectly normal for a lot of code to be required so seldom that it can actually sit in swap for weeks or months without the need of swapping it back in.
Having no swap just means you're wasteing memory.
taltamir wrote:As for open sourcing...
While open sourcing something allows people to more easily develop exploits, it also allows more people to CATCH exploits. Thats why open source browsers (mozilla, etc) are SAFER then closed source ones (internet explorer) despite the funding difference. Yes if anidb is opened sourced people could more easily develop exploits, specifically, they could try to abuse the autoreport feature of AOM to mess up the database. So keep PARTS of the AOM closed source, specifically, the parts regarding authentification of the AOM client and autoaccepting the change requests made by it would be wise (although it will still be possible to crack them, but a good hacker could do it right now!). By having all the LOYAL members of anidb with skills in cracking and security and programming help anidb will actually become more secure then ever, while at the same time have many more features and have alot of bugs expunged. It has been proven to work time and time again with various open source projects!
I really don't think it is that easy. I've talked to quite some people who were initially interested in helping out. But the mayority looses interest pretty soon and never gets to deliver anything.
However, as I already said elsewhere it is very likely that we will reach a point where we need to start a complete rewrite of anidb. At that point in time the open source issue may get a lot more interesting.

BYe!
EXP
FireEye
Posts: 2
Joined: Thu Mar 25, 2004 6:40 pm

Post by FireEye »

Well... as i said i can host it for you... got all needs...

Intel Pentium 2.3Ghz
1024 Mb Ram
10/10 Mbit BredBandsBolaget (sweden)

also said to mail me if you are interested... but got no respond ~~...

*sry if this count as a doubble post*
nwa
AniDB Staff
Posts: 585
Joined: Sat Jun 07, 2003 10:51 am

Post by nwa »

@ FireEye
exp wrote:If you would be willing to provide free hosting for AniDB please send me a private message on the forum.
As you can read, you should send exp a PM in this forum, also provide some more info like is the server located in a datacenter or at your home, what OS is on it, HDD space, do you have full root access and that nothing else is running on that server... all that info you should add to the PM of course. :P
zloy
Posts: 1
Joined: Fri Nov 25, 2005 9:33 pm
Contact:

Post by zloy »

Hi, my name is Nik, I'm from Russia. I came here just to tell that I miss anidb.info. Thank you for it, I really love this site! I hope that it will be online again soon.
Good luck!

excuse me my english pls! :wink:
Syco54645
Posts: 4
Joined: Fri Nov 11, 2005 2:24 am

Post by Syco54645 »

this is a bit off topic, but a nice way to resolve a forked process problem. in linux one can simply set a ulimit (the man page can be found at http://wwwcgi.rdg.ac.uk:8081/cgi-bin/cg ... n/2/ulimit )

dont know if this was done, just thought i would mention it though

-Syco54645
taltamir
Posts: 36
Joined: Wed Mar 23, 2005 4:53 am

Post by taltamir »

so if a peice of info sits on the swap for monthes without being accessed... why is it loaded in the first place? It could have been easily accessed from the harddrive (which it IS, thats what the swap is, harddrive!) once its actually needed. Its already on the harddrive, loading it to swap just makes a copy of it from one point in the HDD to another point in the HDD.

It might be better for a safety concern though, to not let any program mess with the files on the HDD (where a badly written program could corrupt files its not supposed to touch, or maybe to protect files from corruption due to a power failure by first making a copy and then tweaking the copy rather then the original), but rather keep it in a "sandbox" on the HDD.

There are quite alot of programs btw that DO use the harddrive. The game neverwinter nights for example loads the current module into a current module folder in its home directory. MS programs and installations also use the HDD directly rather then swap rather often (office temp files, office / windows update installations, etc).



I do have to admit though that I DON'T know how effective linux is in controlling swap in comparison to windows. It might be alot better. Although some people here said its really bad at it.
derobert
Posts: 4
Joined: Tue Apr 26, 2005 11:50 pm
Contact:

Post by derobert »

taltamir wrote:so if a peice of info sits on the swap for monthes without being accessed... why is it loaded in the first place?
There are several reasons. There are often programs running which have been sleeping for days, weeks even (for example, gettys on the virtual consoles, if you don't use them to log in routinely).

Another example: When programs leak memory. Since the memory is leaked, the program does not access it anymore. However, the kernel can't know that its leaked, so it needs to keep the contents available just in case. That winds up getting swapped out.

Another one, really evil, is non-PIC shared libraries.
VRWarper
Posts: 1
Joined: Sun Nov 27, 2005 2:05 am
Contact:

Post by VRWarper »

If power failure is a problem, then you should have some sort of back up power ^^'. Besides, critical files shouldn't be changed often enough such that a write can be interrupted and corrupt the file.

As for swap, the peice of info that sits in swap might be used only once every few months or it might be used every other minute. What will you do with such an info?

The problem with swap on windows is the fact that the swap lives on the same partition as everything else. On linux, swap gets its own partition. I suppose a problem with such an approach is the fact that swap cant just grow at will like it can on windows. On the other hand, swapping things in and out does not lead to fragmentation of the other partitions like it would on a windows spstem.

As far as preventing badly written programs from touching files an a harddrive, how do you propose telling a bad software from a good one? Some certification? What if the good program happen to have a bug that makes it bad?

Your discription of copying a file before editting and so on... It's like keeping a history of a file right? If the most recent history is corrupt, revert to the last one. That sounds like what a journaling file system does :]
taltamir wrote:so if a peice of info sits on the swap for monthes without being accessed... why is it loaded in the first place? It could have been easily accessed from the harddrive (which it IS, thats what the swap is, harddrive!) once its actually needed. Its already on the harddrive, loading it to swap just makes a copy of it from one point in the HDD to another point in the HDD.

It might be better for a safety concern though, to not let any program mess with the files on the HDD (where a badly written program could corrupt files its not supposed to touch, or maybe to protect files from corruption due to a power failure by first making a copy and then tweaking the copy rather then the original), but rather keep it in a "sandbox" on the HDD.

There are quite alot of programs btw that DO use the harddrive. The game neverwinter nights for example loads the current module into a current module folder in its home directory. MS programs and installations also use the HDD directly rather then swap rather often (office temp files, office / windows update installations, etc).



I do have to admit though that I DON'T know how effective linux is in controlling swap in comparison to windows. It might be alot better. Although some people here said its really bad at it.
rescuetoaster
Posts: 4
Joined: Thu Nov 24, 2005 4:34 pm

Post by rescuetoaster »

Hi!

What i do not get is why to put all your eggs in the same basket... if server failes or admin is wasted then everything is down.
why not build a cluster out of many servers worldwide... thisway u also disperse the resource issue...

EDIT: and have more redundancy

RT.
egg
Posts: 769
Joined: Tue Nov 11, 2003 7:17 am

Post by egg »

rescuetoaster wrote:What i do not get is why to put all your eggs in the same basket... if server failes or admin is wasted then everything is down.
why not build a cluster out of many servers worldwide... thisway u also disperse the resource issue...
What you forget is the basket is free... NOBODY related to AniDB receives money for the project. One person has graciously donated usage of a server. Unless you can find many people to donate usage of servers to make a cluster, this is not even conceivable (never mind technical issues). As it is AniDB is asking for help finding an alternative for hosting, if you or someone you know can help then let us know.

Remember AniDB is FREE without ads!!! Other sites that would provide the same range of functionality would mostly be subscription based or blanketed with advertisements.

AniDB so far has not had to depend on donations, and doesn't want to become dependent. There is the possibility to move to a PAID server, but that would require money (I got a quote that was about $150 US/month for the specified configuration). That could be covered by donations (or ads), and I am sure that people would be willing to pay and things would probably be OK at first. The issue is later on, at some point they will probably be short of $$$. Then they would have to start begging for money or find an alternative.

The general sentiment so far is they want to keep things free. There will be issues from time to time (even if every suggestion in this topic is doable and is implemented, not EVERY scenario is predictable and there will be down time in the future), but it is better to stick with a free service than it is to resort to subscriptions, donations or ads.

As a final point, although there is no redundancy in the server itself, there is some redundancy for the site, the forum and wiki are [obviously] on a separate server, so they are not affected by the problem that is being experienced. This does allow for updates and information to be posted even though the server is not functioning.
Der Idiot
AniDB Staff
Posts: 1227
Joined: Fri Mar 21, 2003 10:19 am

Post by Der Idiot »

UPDATE #3:
ANIDB WILL BE BACK UP IN THE NEXT FEW DAYS!
We found a new hosting that fits our needs. further details will follow once anidb is back up.
rescuetoaster
Posts: 4
Joined: Thu Nov 24, 2005 4:34 pm

Post by rescuetoaster »

I see I was not understood.

Im sure there are more than one (me included) people who could offer quarter of needed resources that are at the current moment needed.
So why not try to solve the problem as cluster or sattelite system... i would not mind post your database on my company (my wery own) server. the thing is that you ask to mutch for me... so why not divide the load... lets say 6-8 people or more... let them house each a part of whole thing and bring it all together in one single site?

Why it MUST BE a single server?! Explain to me ... the dumb....

RT
Last edited by rescuetoaster on Sun Nov 27, 2005 11:08 am, edited 1 time in total.
rescuetoaster
Posts: 4
Joined: Thu Nov 24, 2005 4:34 pm

Post by rescuetoaster »

Der Idiot wrote:UPDATE #3:
ANIDB WILL BE BACK UP IN THE NEXT FEW DAYS!
We found a new hosting that fits our needs. further details will follow once anidb is back up.
Praise the lord and the provider!

RT
Klutz
Posts: 42
Joined: Sun Jun 06, 2004 5:59 am

Post by Klutz »

Ahh, great news. Thank you mr. Provider, thank you AniDB. Now I just need to watch the new Hellsing and School Rumble OVAs and my X-mas is complete.
Locked