AniDB DOWN - NOW UP AGAIN!

All your questions about AniDB belong in here.
No download support!

Moderator: AniDB

kamio
Posts: 2
Joined: Wed Nov 23, 2005 12:44 pm

Frequently accessed like Google; free mirroring

Post by kamio »

Klutz wrote:Funny how addicted one can get to a website, taking it's presence for granted. Stuff like this is bound to happen at some point...just remember guys how much your site and work is appreciated by thousands. :)
Yeah, I can relate to that myself. At first, I saw the site as a novelty, but in just a few short months, I used AniDB to rack up hundreds of gigs of downloads, and I use it daily to manipulate mylist and research on new files to get.

By the way, would it be possible to allow free mirroring of AniDB's content?
I know it's hard because the right to make copies will have to be granted, but for example, Wikipedia allows it (dumps can be found at http://download.wikimedia.org/).
derobert
Posts: 4
Joined: Tue Apr 26, 2005 11:50 pm
Contact:

Post by derobert »

Couple of questions: First, how good a server are we talking about, and how much bandwidth?

Second, if this is linux, try setting up the "watchdog" package, even if you don't have a hardware watchdog on the board --- use the 'softdog' driver. Usually (almost always for the hardware watchdog) it'll reset the server for you when all hell breaks loose.

Third, several companies make remote-reboot power outlets which you can use to hard power cycle the server via SNMP, HTTP, etc.
MilesMi
Posts: 13
Joined: Fri Jan 07, 2005 11:18 pm

Open sourced database?

Post by MilesMi »

kamio wrote:By the way, would it be possible to allow free mirroring of AniDB's content?
I know it's hard because the right to make copies will have to be granted, but for example, Wikipedia allows it (dumps can be found at http://download.wikimedia.org/).
I was concerned about this since the time I found AniDB, but it is written somewhere that they won't give you access to the database, no matter how much you ask.

This is somewhat relevant, since everything on AniDB is the work of its users, AniDB would be nothing without the contribution of them. Yet, it is so useful that people don't think twice before contributing, and together they all built this repository that become so important to our lives that we miss so much now that it is down.

I'm an open source developer, and so, I'm very inclined to the "openess" of the information. I appreciate very much how Wikipedia works. "Your contributions, as well as everyone else ones, are all yours too."

This should also be a concern to the developers. If tomorrow or the day after we get another AniDB-like service, with some important feature that AniDB doesn't have (like a real open source model), everyone will move on. You may think that this would require too much effort from users, so you have a place granted... don't underestimate, people DO change if they find something valuable on your competition, even if the new one doesn't match yours in quality. There are so many real examples: Windows->Linux, Java->.Net, Perl->PHP, Postgresql->MySQL, "Cathedral"->"Bazar", etc...

You should not fear having forks of AniDB if you go open, unless you don't keep improving in quality (which will become easier if you go open). For example, there are loads of technical and usability issues in AniDB website that I could report or even offer myself to fix, but I don't feel like since it is not open, and there are so few people working on it (some suggestions would require a major rewrite of the interface code).

On a related issue, something else is concerning me: how is AniDB data backuped? Realtime DB replication? Or a once-a-day offsite backup? Do you really have nice dreams every night because you know that the database is safe?

I work as a network systems administrator (the kind that would get fired if let the website down for more than 60 minutes, and this makes me feel very uneasy seeing AniDB down for so much time). I don't know how did you assume that apache is stuck in a loop, but the symptoms I was observing as a normal user takes me to other conclusions:

* The server (still accessible via anidb2.ath.cx) answers to ping, with a low delay.
* Apache is accepting connections, but doesn't answer them.
* Exp said SSH is refusing connections.

Unless there is something more I am ignoring or that I don't know, this doesn't look like an infinite loop taking system resources. Even a fork bomb wouldn't give these symptoms. This looks more like some kind of kernel panic (since it is still responding to ping and nothing more is working), in a worst case scenario, caused by your harddisks or disk controller going bye-bye.

Since this server is not well-maintained, I would suggest installing some kind of software watchdog, it would prevent some situations where you get locked out of your box.
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Post by exp »

spankydam
Posts: 15
Joined: Sat Feb 15, 2003 3:26 pm
Location: UK (still waiting for ride off this planet)

Post by spankydam »

Many thanks for the update, here's hoping it can be fixed by a power cycle and it's nothing more serious
exp
Site Admin
Posts: 2438
Joined: Tue Oct 01, 2002 9:42 pm
Location: Nowhere

Re: Open sourced database?

Post by exp »

MilesMi wrote:how is AniDB data backuped?
once a day, off site.
MilesMi wrote:I don't know how did you assume that apache is stuck in a loop
Experience, we had the exact same problem a couple of times already.
Most of the time it is harmless, as long as someone logs in within a couple of hours and restarts the apache process. Everything goes back to normal again. But the system load and process count increases with every hour so if you take too long the system reaches a state where it is no longer possible to login.
And yes, I installed a software watchdog last time, but for some reason it does not seem to take action x_X
I coded some other stuff now which I will install once the server is back up.
And for those who like numbers:
this is the current situation @ server
sysload: 674.82 412.29 574.67 - processes running: 4147/11685 - swap used: 997 MB
MilesMi wrote:in a worst case scenario, caused by your harddisks or disk controller going bye-bye.
I am strongly suspecting mod_throttle so far.
MilesMi wrote:Since this server is not well-maintained, I would suggest installing some kind of software watchdog, it would prevent some situations where you get locked out of your box.
Problem is that I would need to update the kernel to really use that. And I don't dare to do that, because the system might no longer boot up for good then. And as you can see the response times of the local admin are pretty bad :P

About the open source part, well, we might get to that point someday. A total rewrite of anidb will be needed at some point. But for now we believe that keeping the source and the db closed is for the better.
I think there was a length discussion about this in this forum somewhere too.

BYe!
EXP
PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW »

Well, one of the major reasons to keep it closed source is to protect user data. The number of people who have access to everyones file list can be counted on one hand, aka if you want to keep your mylist private it's private (this is very important to many people).
MilesMi
Posts: 13
Joined: Fri Jan 07, 2005 11:18 pm

Re: Open sourced database?

Post by MilesMi »

exp wrote:And yes, I installed a software watchdog last time, but for some reason it does not seem to take action x_X
I coded some other stuff now which I will install once the server is back up.
Note if you are going to write your own watchdog: it should run with higher priority, possibly "realtime" scheduling, be small, and should detect its own unexpected behaviour and commit suicide if this happens. Also, must handle SIGSEGV (which is not simple, since your stack and heap may be f***ed up).
exp wrote:And for those who like numbers:
this is the current situation @ server
sysload: 674.82 412.29 574.67 - processes running: 4147/11685 - swap used: 997 MB
Err... how did you get this information without shell or web access?

So something is varying... in the last 5 minutes we had less load than in the last 15 minutes, and then, the last 1 minute we had more load than the last 15 minutes. I couldn't understand the two numbers of processes... perhaps current and maximum?

But your swap is way too big, and here we have a real problem. Looks like you have about 1GB of swap, and this is what is limiting your system load. 1GB is too much swap for our current harddisk technology. People tend to use that outdated rule that you should have twice swap space as you have RAM. This is no longer true since RAM modules are getting faster and larger faster than HDs are getting faster.

Do yourself a small experience: write a 1GB file to your disk, then read it again, and measure how much time it gets (you must already know presuming that you regularly hash your files). Now imagine this being used as virtual RAM space for a couple of processes that allocated too much memory in an unpredictable way, being swapped in and out all the time... This can surely bring your system down. Linux is pretty stupid swapping... if you give it enough cord, it will hang itself on it.

Currently, I recommend you to think twice if you are going to setup more than 256MB or 512MB of swap. A swap of 1GB is only suitable for >=2GB RAM, if you really need it and have a good SCSI 160MB/s HD. A swap of 2GB only for >=8GB RAM on a RAID-1 array with 320MB/s throughput. Even in these cases, you have to have a good control on the behaviour of your processes (tip: use ulimit).

On all other cases, having a small swap is better. In a swap-intensive situation, where processes allocate memory out of control, they will fill the swap in less time, and get killed quicker without raising the system load. Also, with smaller swap space, the harddisk heads won't have to move too much on a swap-intensive situation, so not degrading system performance that much.

Finally, now that you said that the swap is full, and I presume you are on a swap-intensive situation with a high system load, for more than 24 hours, you have yet another problem: hardware stress. Even the most tough harddisks don't survive for too long if writing constantly to the same region of the disk for so much time. The magnetic surface of the disk degrades a little tiny bit on each rewrite (bad sectors being the result in the long run).

Even if you can't reboot it, if you know a way to turn it off until the admin gets back (i.e. call his mother and ask her to turn of his room's circuit breaker, whatever...) is better than letting it in the current state.
exp wrote:Problem is that I would need to update the kernel to really use that. And I don't dare to do that, because the system might no longer boot up for good then. And as you can see the response times of the local admin are pretty bad :P
You know that currently you have the option to run a Linux inside another, right? Although it is not easy to get it right, it gives you higher control on what looks like virtual servers, inside a single machine.

Best regards.
atreya2011
Posts: 12
Joined: Tue Jun 21, 2005 11:26 am

Post by atreya2011 »

I was able to add files to mylist using AniDB O Matic, even when AniDB was down. Is that normal?
PetriW
AniDB Staff
Posts: 1522
Joined: Sat May 24, 2003 2:34 pm

Post by PetriW »

atreya2011 wrote:I was able to add files to mylist using AniDB O Matic, even when AniDB was down. Is that normal?
Yes, that's normal. However, downloads from the api will not work as they rely on the apache.
kamio
Posts: 2
Joined: Wed Nov 23, 2005 12:44 pm

Post by kamio »

PetriW wrote:Well, one of the major reasons to keep it closed source is to protect user data. The number of people who have access to everyones file list can be counted on one hand, aka if you want to keep your mylist private it's private (this is very important to many people).
Just to clarify (which I didn't think was needed), I was referring to public data being open, not private. That is, the material that is useful as reference, including file info, anime, groups, etc.

Private profiles are generally not useful to people, and plus, mirroring them would be a privacy issue. (In fact, with the public reference data, people can even build their own user management systems.)

Thanks =)
shaman
Posts: 23
Joined: Tue Feb 15, 2005 9:01 pm
Location: Vilnius, Lithuania
Contact:

Post by shaman »

Where is the server situated geographically? Maybe some dedicated AniDB users might help rebooting it? (I suppose it's not in somekind of a datacenter or smth. so getting physical access to it might be relatively easy (e.g. to it's circuit breakers ;) ))
tub0rg
Posts: 1
Joined: Wed Nov 23, 2005 6:44 pm

Post by tub0rg »

Where is the server situated geographically? Maybe some dedicated AniDB users might help rebooting it? (I suppose it's not in somekind of a datacenter or smth. so getting physical access to it might be relatively easy (e.g. to it's circuit breakers ))
i thought so too... if you guys know where your server admin lives.. someone could ring at the door and ask him nicely to reboot that thing...
shaman
Posts: 23
Joined: Tue Feb 15, 2005 9:01 pm
Location: Vilnius, Lithuania
Contact:

Post by shaman »

Well yeah. I really think if there were some users who live near that place they would gladly help 'cause as i see i'm not the only who can't live without AniDB ;)
Rar
AniDB Staff
Posts: 1471
Joined: Fri Mar 12, 2004 2:41 pm
Location: UK
Contact:

Post by Rar »

The server is located on a small geostationary satellite. In the past we've had cosmonauts from the Mir space station walk across and hard reboot it, but that's somewhat scuppered now, and the ISS people are not nearly as obliging.

Rar
Locked