Friday, April 18, 2014

Downtime weekend of 4/19/2014

UPDATE 04/20/14-11:40AM EDT

We have started the update for the database server so will be offline for at least several hours. I will be monitoring the situation and get everything back up as soon as possible. 

UPDATE 04/20/14-2:00AM EDT

First round of updates went smoothly with no downtime or issues. Tomorrow's update will cause downtime though. Probably 3-4 hours.

ORIGINAL POST:

Linode, our hosting service, has upgraded their services and in order to take advantage of the upgrades we need to power down our machines. This first phase of this upgrade will occur on Saturday 4/19/2014, and will involve upgrading our frontend and apache clone. This should not require taking the site offline but could potential cause strange behavior or decreased performance.

The second phase will occur at 12:00pm EDT Sunday, 4/20/2014 and will require shutting down our database server. This will cause the site to go offline. We estimate that this should take 3-4 hours. 

For more information on the upgrades see Linodes blog post.

Sunday, April 13, 2014

Connectivity issues at our hosting facility

The Newark hosting facility where our servers are located is experiencing connectivity issues. This does affect the RationalWiki websites, status updates are available here.

Friday, February 28, 2014

Incoming! Rapid server reconfiguration underway ...

RationalWiki just got sued by Kent Hovind. The lawsuit is a comedy classic and will brighten your Friday afternoon. (Saloon Bar discussion.)

We're not worried about the suit — the worry is that we're about to get a zillion well-wishers hammering the server. Trent is frantically rejigging our setup as we write this, pressing a spare box into service. More news as it comes, and if the site entirely vanishes please try again five minutes later ...

Update, next day: We quickly spun up a 4GB Linode to just put Apache on (MediaWiki scales horizontally really well). It appears that Apache was taking lots of CPU, not MySQL! So we've split Apache for the main site between the two boxes (the old 8GB and the new 4GB) and the setup is ticking along nicely. Trent may have some nice graphs later.

Wednesday, February 5, 2014

The wiki is getting hammered.

Haven't checked the Squid logs as to what it is —I suspect fallout from the Nye/Ham debate. But we're getting a buttload of requests, a lot of Apache processes, load around 50-55, MySQL using 70% of CPU after a restart ... we're not actually running out of memory, the wiki's just legitimately doing a lot of work. So page loads are often working, but taking up to a minute, and there's lots of timeouts. David's keeping an eye on stuff.

Edit: It doesn't look like any particular article. Our creationism stuff in general is getting lots of hits. While youre waiting, here's today's Google cache of "How come there are still monkeys?"

Edit 2: Still very highly loaded, but is serving up pages now, if very slowly.

Sunday, July 28, 2013

Apache zombie process problems.

The Apache server is being a dick. It keeps dying with unkillable zombie processes; these don't serve data, but they do keep their hold on port 80. The only workaround is to reboot the server. This is, of course, ridiculous. I've put a ticket in with Linode. Anyone got ideas? It's just the standard Ubuntu apache2 package.

Update: Apache was causing kernel oopses. (Things that make you go "wtf.") Linode suggest using kernel 3.9-linode instead of 2.6-linode (the default for Ubuntu 10.04, which our box runs on). We'll see how that goes next reboot. Might even fix the white-screening blog too.

Friday, July 12, 2013

Wiki slow, we're being hammered.

The wiki has been really slow all day. The Apache server and the Squids appear quite happy with life - but the load balancer is reporting huge traffic spikes. More information as I have it.

(The blogs are unaffected, as they're served directly from the Apache server.)

Update: Here's the load balancer connections per second. (The gaps are when logging is killed to increase throughput or something.) Note the bit where we go from a few tens of queries a second up to 800 a second.

Monday, May 27, 2013

Site is doing its yoyo impression again.

I'm not sure what's going on (and I'm currently without broadband, so I don't have reliable internet time to diagnose the problem), but the Apache server is occasionally going into conniptions and needing a reboot. (It's in one as I speak.) We are aware of the problem ... fwiw.