Posted: Fri Nov 25, 2005 6:11 am
well, MilesMi really nailed everything. One question though, how much ram do you have? With sufficient ram its better to completely eliminate swap, or set an autoloading ramdrive and put the swap on it for badly written applications (applications that use swap instead of ram).
Just to clarify, Swap in itself is a really bad idea. A good program just access' its files directly, and caches things to ram for increased speed. Swap is using the HDD as ram AS A FALLBACK when you are OUT OF RAM to prevent the computer from crushing from a program trying to claim more ram then there is available (again, badly written program). Ideally though, if data needs to be written to the HDD its better to write it directly rather then writing it as swap.
Proper sequence is:
1. HDD to RAM for caching
2. RAM to CPU for processing backto RAM.
3. repeat 2 while processing a certain peice of data. (well, L2 cache etc are all just ram blocks found at closer distances to the CPU)
4. Use completed calculations, or forward them to a dedicated hardware peice (audio/video) or save to HDD
However, if you run out of ram this sequence will look like this:
1. HDD to RAM for caching, not enough RAM, some cached back to swap (data1) on HDD
2. RAM to CPU for processing of some data (data2).
3. RAM to Swap (data2) to free more RAM.
4. Swap (data1) to RAM
5. RAM to CPU for processing of some data (data1).
6. repeat 2 through 5 until calculation complete.
7. Use completed calculations, or forward them to a dedicated hardware peice (audio/video) or save to HDD
This is obviously meant to be only done in emergency since you are copying from one point on the harddrive to another and then processing that. A practical example would be to get 7z compression tool and try to compress with a dictionary thats too large. If its borderline, you will not the process continuing until you run out of ram, then it starts using swap and suddenly your harddrive starts making noise like crazy while 7z seems to be "stuck". However, if you are trying to compresss a REALLY small file (a few meg) and have a powerful computer, and leave it on for a LOOOONG time you will notice it slowly moving (it might go 1% per hour of operation or some such).
Just to clarify the speed differences. HDD's access time is measures in miliseconds (1/10^3) while ram access speed in 1/10^9. And most HDD have an access time of about 100ms (thats 1/10 of a second). This is a VERY long time for random access, harddrives are much better as SEQUENTIAL transfers (ie, they might be able to transfer a file at 100mb/s)
An example of a badly written program, "spellforce: the order of dawn" needs about 2GB of swap space to avoid random crushes, REGARDLESS of how much ram you have (It doesn't use that much swap space, but it will sometimes try to write to a swap address thats very high, and if you dont have sufficient swap space it will fail to write it because it does not exist). Oh and it uses all that swap space while NEVER using more then 300MB of ram (tested on a 512MB and a 1GB of ram machine).
A situation where using swap will prvevent a crash and will NOT cause your computer to stop responding is if for example you have 500MB of ram, and your OS's user interface takes 100MB, and you run a program that takes 450MB of ram, that program will load into ram forcing parts of the OS into swap, those parts will stop responding (where windows aren't moving, mouseclicks aren't responded to, you know the drill) while the program's thread is being processed. Once said thread is done it will terminate and the OS portions is swap move back into the ram and come back to proper speed.
If said program is NOT stuck in a loop but takes a really long time to finish your computer might look frozen, but will "unfreeze" if you wait a couple hours.
If the program requires parts of the OS that were pushed to swap to complete its operations then it could literaly push the completion time to thosands of years, in which case it's completely frozen, not to mention it will kill your HDD before it finishes.
In fact, most often it is NOT an infinate loop that causes a "freeze" but rather misuse of swap, which windows handles in a very stupid manner.
As for open sourcing...
While open sourcing something allows people to more easily develop exploits, it also allows more people to CATCH exploits. Thats why open source browsers (mozilla, etc) are SAFER then closed source ones (internet explorer) despite the funding difference. Yes if anidb is opened sourced people could more easily develop exploits, specifically, they could try to abuse the autoreport feature of AOM to mess up the database. So keep PARTS of the AOM closed source, specifically, the parts regarding authentification of the AOM client and autoaccepting the change requests made by it would be wise (although it will still be possible to crack them, but a good hacker could do it right now!). By having all the LOYAL members of anidb with skills in cracking and security and programming help anidb will actually become more secure then ever, while at the same time have many more features and have alot of bugs expunged. It has been proven to work time and time again with various open source projects!
Just to clarify, Swap in itself is a really bad idea. A good program just access' its files directly, and caches things to ram for increased speed. Swap is using the HDD as ram AS A FALLBACK when you are OUT OF RAM to prevent the computer from crushing from a program trying to claim more ram then there is available (again, badly written program). Ideally though, if data needs to be written to the HDD its better to write it directly rather then writing it as swap.
Proper sequence is:
1. HDD to RAM for caching
2. RAM to CPU for processing backto RAM.
3. repeat 2 while processing a certain peice of data. (well, L2 cache etc are all just ram blocks found at closer distances to the CPU)
4. Use completed calculations, or forward them to a dedicated hardware peice (audio/video) or save to HDD
However, if you run out of ram this sequence will look like this:
1. HDD to RAM for caching, not enough RAM, some cached back to swap (data1) on HDD
2. RAM to CPU for processing of some data (data2).
3. RAM to Swap (data2) to free more RAM.
4. Swap (data1) to RAM
5. RAM to CPU for processing of some data (data1).
6. repeat 2 through 5 until calculation complete.
7. Use completed calculations, or forward them to a dedicated hardware peice (audio/video) or save to HDD
This is obviously meant to be only done in emergency since you are copying from one point on the harddrive to another and then processing that. A practical example would be to get 7z compression tool and try to compress with a dictionary thats too large. If its borderline, you will not the process continuing until you run out of ram, then it starts using swap and suddenly your harddrive starts making noise like crazy while 7z seems to be "stuck". However, if you are trying to compresss a REALLY small file (a few meg) and have a powerful computer, and leave it on for a LOOOONG time you will notice it slowly moving (it might go 1% per hour of operation or some such).
Just to clarify the speed differences. HDD's access time is measures in miliseconds (1/10^3) while ram access speed in 1/10^9. And most HDD have an access time of about 100ms (thats 1/10 of a second). This is a VERY long time for random access, harddrives are much better as SEQUENTIAL transfers (ie, they might be able to transfer a file at 100mb/s)
An example of a badly written program, "spellforce: the order of dawn" needs about 2GB of swap space to avoid random crushes, REGARDLESS of how much ram you have (It doesn't use that much swap space, but it will sometimes try to write to a swap address thats very high, and if you dont have sufficient swap space it will fail to write it because it does not exist). Oh and it uses all that swap space while NEVER using more then 300MB of ram (tested on a 512MB and a 1GB of ram machine).
A situation where using swap will prvevent a crash and will NOT cause your computer to stop responding is if for example you have 500MB of ram, and your OS's user interface takes 100MB, and you run a program that takes 450MB of ram, that program will load into ram forcing parts of the OS into swap, those parts will stop responding (where windows aren't moving, mouseclicks aren't responded to, you know the drill) while the program's thread is being processed. Once said thread is done it will terminate and the OS portions is swap move back into the ram and come back to proper speed.
If said program is NOT stuck in a loop but takes a really long time to finish your computer might look frozen, but will "unfreeze" if you wait a couple hours.
If the program requires parts of the OS that were pushed to swap to complete its operations then it could literaly push the completion time to thosands of years, in which case it's completely frozen, not to mention it will kill your HDD before it finishes.
In fact, most often it is NOT an infinate loop that causes a "freeze" but rather misuse of swap, which windows handles in a very stupid manner.
As for open sourcing...
While open sourcing something allows people to more easily develop exploits, it also allows more people to CATCH exploits. Thats why open source browsers (mozilla, etc) are SAFER then closed source ones (internet explorer) despite the funding difference. Yes if anidb is opened sourced people could more easily develop exploits, specifically, they could try to abuse the autoreport feature of AOM to mess up the database. So keep PARTS of the AOM closed source, specifically, the parts regarding authentification of the AOM client and autoaccepting the change requests made by it would be wise (although it will still be possible to crack them, but a good hacker could do it right now!). By having all the LOYAL members of anidb with skills in cracking and security and programming help anidb will actually become more secure then ever, while at the same time have many more features and have alot of bugs expunged. It has been proven to work time and time again with various open source projects!