Page 3 of 7
Posted: Wed Nov 23, 2005 10:42 pm
No wonder it's hard to reach
It must have sort of high ping times if so
I've remembered MilesMi solution to save hard drives the one with "call his mother and tell her to turn off the circuit breaker" i can already imagine how that should look like
Re: Open sourced database?
Posted: Thu Nov 24, 2005 12:09 am
exp wrote:But the system load and process count increases with every hour so if you take too long the system reaches a state where it is no longer possible to login.
Put ulimit -H -u 300 (for example) in Apache's init script, make 300 some sane number. It sounds very much like you're describing a fork bomb....
And, btw, you might be able to ssh in. How far does ssh get? If you're getting past authentication, doing something like: ssh root@server 'killall -9 apache' might work.
And yes, I installed a software watchdog last time, but for some reason it does not seem to take action x_X
If its the normal watchdog package, the reason is probably that you didn't configure it to require a certain number of free process slots, load average < x, etc. Settings for these are in /etc/watchdog.conf.
Also, make sure you have uncommented the /dev/watchdog line in that config file, too, or its not actually doing anything!
Problem is that I would need to update the kernel to really use that. And I don't dare to do that, because the system might no longer boot up for good then. And as you can see the response times of the local admin are pretty bad
If you have loadable module support enabled, you don't need to compile a new kernel.[/quote]
Re: Open sourced database?
Posted: Thu Nov 24, 2005 12:54 am
MilesMi wrote:But your swap is way too big, and here we have a real problem. Looks like you have about 1GB of swap, and this is what is limiting your system load. 1GB is too much swap for our current harddisk technology.
Bull. The amount of swap used is utterly irrelevant (so long as you're not out!); what matters is the amount of thrashing. Having 1GB of leaked memory sitting in swap doesn't really hurt anything. Having a single megabyte constantly thrashing in and out of swap will kill performance.
On all other cases, having a small swap is better. In a swap-intensive situation, where processes allocate memory out of control, they will fill the swap in less time, and get killed quicker without raising the system load.
First off, limiting memory usage is much better done with ulimit. Second, this depends on your system's usage --- if you have a program which, for example, slowly leaks memory, having a lot of swap is perfectly appropriate (of course, fixing the memory leak would be better, but we can't always do that).
Posted: Thu Nov 24, 2005 1:34 am
i'll try to clear up some issues.
1) the server is located in a datacenter and reboot is done by a call to the 24h support staff IIRC. Problem is that I neither have the number nor the credentials to call in for a reboot :|
2) we need the swap space. AniDB is running some VERY heavy cron jobs a couple of times a week which lead to extensive swapping. In terms of disks, we're talking about an scsi hardware raid with two disks (mirrored) here.
3) ssh - ssh doesn't even get as far as the authentication
4) watchdog/ulimit - i will look into that again once the server is back up
5) how to get load info/add stuff to mylist via AoM - apache is broken and the system load is extremely high. But the database, the bots and the APIs do still work (if very slowly). Which means a certain amount of interaction with anidb is still possible. And the irc bots can provide the load info.
6) process count numbers: the first number is the number of processes currently waiting for cpu cycles and the second number the total number of processes currently running.
Posted: Thu Nov 24, 2005 2:45 am
so everything is running ont he same server...
hmm, haven't you thought of something in the lines of VMWare ESX server?, you could always log to the VMware manager and hard reboot the virtual server...
drawbacks: ESX is the top-of-the-line virtual server software there is, last time i checked, ~2400u$s for a 2-CPU licence
Is there some resource partitioning software for linux?, i know Win2003 Enterprise has a free MS download that does resource partitioning, with that you'll never end up with a process that kills the system.
and this is how far my help can go as i have 0.6 experience on Linux administration
Posted: Thu Nov 24, 2005 3:55 am
there are actually multiple ways of doing this under linux.
i.e. via user mode linux or vmware workstation (which is a lot cheaper :P).
but the main drawback here is that you loose a lot of performance.
and performance is very critical for anidb :|
therefore I don't think those approaches are viable.
Posted: Thu Nov 24, 2005 4:20 am
you know guys.... im pretty sure that the anidb staff knows what works best in this specific situation, based on funding, which afaik is ziltch. meaning they are working on what someone donates. 1gb of swap is not insane, infact on my busy webserver i have 3gb of swap. my personal thought, dont tell someone how to do their job. im pretty sure the can deal with this. and to mister smartass "im worried cause the box has been down for so long zomg i cant get my anime" network systems administrator, take a chill pill dude. anidb is free. dont bitch.
k thx bye. all complaints on my post can go to email@example.com
So late news...
Posted: Thu Nov 24, 2005 10:07 am
Sry to say this, I think something wrong with my pc or network. I swearing and hating this happen when I couldn't view the sites. I think it was about around 2 weeks I trying to archieve it, isnt? Or it just happen to me alone?
Gladly today I was about trying to visit the sites again, or I will kill myself...
Arghhh... Im a bit down, but thank to all staff for the very usefull information.
I luve ya all!
Posted: Thu Nov 24, 2005 10:38 am
It would be annoying if AniDB went down, ... AnimeNFO isn't my thing, and I don't have a updated back up of my list.
Btw if you ever are going down (forgood or something), please mail people a link on where they can get their dumped mylist.
Some people like me don't have another mylist so if AniDB goes down it'll be very bad lol.
Posted: Thu Nov 24, 2005 10:57 am
i remember the last time the process hanged like this, it took a while to get hold of gmni...
oh well, waiting is the name of the game
Posted: Thu Nov 24, 2005 11:53 am
I think DerIdiot should add a 'Don't PANIC!' line in the first post or something... it has happaned before, it'll come back eventually, one way or another.
Posted: Thu Nov 24, 2005 12:02 pm
Posted: Thu Nov 24, 2005 12:06 pm
get someone to pay you one of those new 1&1 servers they now offer in germany at least with traffic flatrates as it seems, if you find a good financer the 3XL64 for 209€ is a good deal with dual core opteron, 4 gig of ram and a 250 GB RAID1 , the cheaper ones aren't that bad either, at least you could reset those easily if you get stuck in some loops hehe
my tip, if a program ends up in an loop etc. that runs longer than expect, reserve as much memory as you can, usually if it's out of memory, linux should kill the process with the most resources eaten.
Posted: Thu Nov 24, 2005 1:34 pm
plz do your best and keep up the great work.
andb is awesome
Posted: Thu Nov 24, 2005 2:35 pm
Yeah, that's something that paniced me too. I was changing my email address (Softhome.net is dumping the free accounts) , so I change it, get the verification email, and the URL comes up Cannot Find Server or DNS Error. Talk about panic time. Hopefully Anidb will come up soon so I can get this done.