AniDB DOWN - NOW UP AGAIN!
Moderator: AniDB
Re: Open sourced database?
Put ulimit -H -u 300 (for example) in Apache's init script, make 300 some sane number. It sounds very much like you're describing a fork bomb....exp wrote:But the system load and process count increases with every hour so if you take too long the system reaches a state where it is no longer possible to login.
And, btw, you might be able to ssh in. How far does ssh get? If you're getting past authentication, doing something like: ssh root@server 'killall -9 apache' might work.
If its the normal watchdog package, the reason is probably that you didn't configure it to require a certain number of free process slots, load average < x, etc. Settings for these are in /etc/watchdog.conf.And yes, I installed a software watchdog last time, but for some reason it does not seem to take action x_X
Also, make sure you have uncommented the /dev/watchdog line in that config file, too, or its not actually doing anything!
If you have loadable module support enabled, you don't need to compile a new kernel.[/quote]Problem is that I would need to update the kernel to really use that. And I don't dare to do that, because the system might no longer boot up for good then. And as you can see the response times of the local admin are pretty bad
Re: Open sourced database?
Bull. The amount of swap used is utterly irrelevant (so long as you're not out!); what matters is the amount of thrashing. Having 1GB of leaked memory sitting in swap doesn't really hurt anything. Having a single megabyte constantly thrashing in and out of swap will kill performance.MilesMi wrote:But your swap is way too big, and here we have a real problem. Looks like you have about 1GB of swap, and this is what is limiting your system load. 1GB is too much swap for our current harddisk technology.
First off, limiting memory usage is much better done with ulimit. Second, this depends on your system's usage --- if you have a program which, for example, slowly leaks memory, having a lot of swap is perfectly appropriate (of course, fixing the memory leak would be better, but we can't always do that).On all other cases, having a small swap is better. In a swap-intensive situation, where processes allocate memory out of control, they will fill the swap in less time, and get killed quicker without raising the system load.
ok,
i'll try to clear up some issues.
1) the server is located in a datacenter and reboot is done by a call to the 24h support staff IIRC. Problem is that I neither have the number nor the credentials to call in for a reboot :|
2) we need the swap space. AniDB is running some VERY heavy cron jobs a couple of times a week which lead to extensive swapping. In terms of disks, we're talking about an scsi hardware raid with two disks (mirrored) here.
3) ssh - ssh doesn't even get as far as the authentication
4) watchdog/ulimit - i will look into that again once the server is back up
5) how to get load info/add stuff to mylist via AoM - apache is broken and the system load is extremely high. But the database, the bots and the APIs do still work (if very slowly). Which means a certain amount of interaction with anidb is still possible. And the irc bots can provide the load info.
6) process count numbers: the first number is the number of processes currently waiting for cpu cycles and the second number the total number of processes currently running.
BYe!
EXP
i'll try to clear up some issues.
1) the server is located in a datacenter and reboot is done by a call to the 24h support staff IIRC. Problem is that I neither have the number nor the credentials to call in for a reboot :|
2) we need the swap space. AniDB is running some VERY heavy cron jobs a couple of times a week which lead to extensive swapping. In terms of disks, we're talking about an scsi hardware raid with two disks (mirrored) here.
3) ssh - ssh doesn't even get as far as the authentication
4) watchdog/ulimit - i will look into that again once the server is back up
5) how to get load info/add stuff to mylist via AoM - apache is broken and the system load is extremely high. But the database, the bots and the APIs do still work (if very slowly). Which means a certain amount of interaction with anidb is still possible. And the irc bots can provide the load info.
6) process count numbers: the first number is the number of processes currently waiting for cpu cycles and the second number the total number of processes currently running.
BYe!
EXP
-
- Posts: 15
- Joined: Sun Mar 27, 2005 3:32 pm
so everything is running ont he same server...
hmm, haven't you thought of something in the lines of VMWare ESX server?, you could always log to the VMware manager and hard reboot the virtual server...
drawbacks: ESX is the top-of-the-line virtual server software there is, last time i checked, ~2400u$s for a 2-CPU licence
Is there some resource partitioning software for linux?, i know Win2003 Enterprise has a free MS download that does resource partitioning, with that you'll never end up with a process that kills the system.
and this is how far my help can go as i have 0.6 experience on Linux administration
hmm, haven't you thought of something in the lines of VMWare ESX server?, you could always log to the VMware manager and hard reboot the virtual server...
drawbacks: ESX is the top-of-the-line virtual server software there is, last time i checked, ~2400u$s for a 2-CPU licence
Is there some resource partitioning software for linux?, i know Win2003 Enterprise has a free MS download that does resource partitioning, with that you'll never end up with a process that kills the system.
and this is how far my help can go as i have 0.6 experience on Linux administration
there are actually multiple ways of doing this under linux.
i.e. via user mode linux or vmware workstation (which is a lot cheaper :P).
but the main drawback here is that you loose a lot of performance.
and performance is very critical for anidb :|
therefore I don't think those approaches are viable.
BYe!
EXP
i.e. via user mode linux or vmware workstation (which is a lot cheaper :P).
but the main drawback here is that you loose a lot of performance.
and performance is very critical for anidb :|
therefore I don't think those approaches are viable.
BYe!
EXP
you know guys.... im pretty sure that the anidb staff knows what works best in this specific situation, based on funding, which afaik is ziltch. meaning they are working on what someone donates. 1gb of swap is not insane, infact on my busy webserver i have 3gb of swap. my personal thought, dont tell someone how to do their job. im pretty sure the can deal with this. and to mister smartass "im worried cause the box has been down for so long zomg i cant get my anime" network systems administrator, take a chill pill dude. anidb is free. dont bitch.
k thx bye. all complaints on my post can go to sftu@kthxdie.com
k thx bye. all complaints on my post can go to sftu@kthxdie.com
So late news...
Sry to say this, I think something wrong with my pc or network. I swearing and hating this happen when I couldn't view the sites. I think it was about around 2 weeks I trying to archieve it, isnt? Or it just happen to me alone?
Gladly today I was about trying to visit the sites again, or I will kill myself...
Arghhh... Im a bit down, but thank to all staff for the very usefull information. I luve ya all!
Gladly today I was about trying to visit the sites again, or I will kill myself...
Arghhh... Im a bit down, but thank to all staff for the very usefull information. I luve ya all!
It would be annoying if AniDB went down, ... AnimeNFO isn't my thing, and I don't have a updated back up of my list.
Btw if you ever are going down (forgood or something), please mail people a link on where they can get their dumped mylist.
Some people like me don't have another mylist so if AniDB goes down it'll be very bad lol.
Btw if you ever are going down (forgood or something), please mail people a link on where they can get their dumped mylist.
Some people like me don't have another mylist so if AniDB goes down it'll be very bad lol.
-
- AniDB Staff
- Posts: 438
- Joined: Thu Apr 08, 2004 1:43 am
- Location: Portugal
get someone to pay you one of those new 1&1 servers they now offer in germany at least with traffic flatrates as it seems, if you find a good financer the 3XL64 for 209€ is a good deal with dual core opteron, 4 gig of ram and a 250 GB RAID1 , the cheaper ones aren't that bad either, at least you could reset those easily if you get stuck in some loops hehe
my tip, if a program ends up in an loop etc. that runs longer than expect, reserve as much memory as you can, usually if it's out of memory, linux should kill the process with the most resources eaten.
my tip, if a program ends up in an loop etc. that runs longer than expect, reserve as much memory as you can, usually if it's out of memory, linux should kill the process with the most resources eaten.
-
- Posts: 3
- Joined: Wed Mar 02, 2005 8:38 pm
Yeah, that's something that paniced me too. I was changing my email address (Softhome.net is dumping the free accounts) , so I change it, get the verification email, and the URL comes up Cannot Find Server or DNS Error. Talk about panic time. Hopefully Anidb will come up soon so I can get this done.
BelGarion
BelGarion