Incoming! Rapid server reconfiguration underway ...

RationalWiki just got sued by Kent Hovind. The lawsuit is a comedy classic and will brighten your Friday afternoon. (Saloon Bar discussion.)

We're not worried about the suit — the worry is that we're about to get a zillion well-wishers hammering the server. Trent is frantically rejigging our setup as we write this, pressing a spare box into service. More news as it comes, and if the site entirely vanishes please try again five minutes later ...

Update, next day: We quickly spun up a 4GB Linode to just put Apache on (MediaWiki scales horizontally really well). It appears that Apache was taking lots of CPU, not MySQL! So we've split Apache for the main site between the two boxes (the old 8GB and the new 4GB) and the setup is ticking along nicely. Trent may have some nice graphs later.

The wiki is getting hammered.

Haven't checked the Squid logs as to what it is —I suspect fallout from the Nye/Ham debate. But we're getting a buttload of requests, a lot of Apache processes, load around 50-55, MySQL using 70% of CPU after a restart ... we're not actually running out of memory, the wiki's just legitimately doing a lot of work. So page loads are often working, but taking up to a minute, and there's lots of timeouts. David's keeping an eye on stuff.

Edit: It doesn't look like any particular article. Our creationism stuff in general is getting lots of hits. While youre waiting, here's today's Google cache of "How come there are still monkeys?"

Edit 2: Still very highly loaded, but is serving up pages now, if very slowly.

Apache zombie process problems.

The Apache server is being a dick. It keeps dying with unkillable zombie processes; these don't serve data, but they do keep their hold on port 80. The only workaround is to reboot the server. This is, of course, ridiculous. I've put a ticket in with Linode. Anyone got ideas? It's just the standard Ubuntu apache2 package.

Update: Apache was causing kernel oopses. (Things that make you go "wtf.") Linode suggest using kernel 3.9-linode instead of 2.6-linode (the default for Ubuntu 10.04, which our box runs on). We'll see how that goes next reboot. Might even fix the white-screening blog too.

Wiki slow, we're being hammered.

The wiki has been really slow all day. The Apache server and the Squids appear quite happy with life - but the load balancer is reporting huge traffic spikes. More information as I have it.

(The blogs are unaffected, as they're served directly from the Apache server.)

Update: Here's the load balancer connections per second. (The gaps are when logging is killed to increase throughput or something.) Note the bit where we go from a few tens of queries a second up to 800 a second.

Site is doing its yoyo impression again.

I'm not sure what's going on (and I'm currently without broadband, so I don't have reliable internet time to diagnose the problem), but the Apache server is occasionally going into conniptions and needing a reboot. (It's in one as I speak.) We are aware of the problem ... fwiw.

State of the blog: recovering.

And the blog returns from its holiday at the seaside. The blog is now happily resident on the main RationalWiki server. Just waiting for the DNS to resolve everywhere. I also need to fix the front page images (missing because of the way I did the import).
 
The next holiday at the seaside will probably be in a week or two, when I’m busy moving house. But I’m sure our contributors will write in great volume to take up the slack. Just as soon as I get around to recreating their accounts on this instance.

We also have an insanely dull front page, just waiting for a burgeoning blog network full of interesting people to spontaneously manifest from the quantum ether.

State of the blog: addled.

The site the RationalWiki Blog is presently living on is fine for what it is, but what it is includes not being up to our needs, hence the blog's party-stopping yoyo impression for much of today. We're finally getting around to doing the sensible thing and setting it up on the main RationalWiki server (which just got its memory doubled, so there's actually room for stuff). Please excuse hiccups in the meantime.