Downtime for upgrade: Saturday 6 April 2019, 12:00 UTC-14:00 UTC

We get a free upgrade from Linode! We just have to press the button and wait 90 minutes for them to migrate the server.

I'll be pressing the button around 12:00 UTC on Saturday 6 April. Go outside or something while it's happening.

Update, 13:17 UTC: aaaand we're back!

Emergency maintenance, Sun 06 Aug 2017, 0100 UTC on

Linode (our hosting company) has discovered hardware problems, with action to start at 0100 UTC tomorrow morning:

We continuously monitor the health of our equipment, and we've been alerted to an issue with the physical hardware on which your Linode resides. Specifically, there is a hardware issue with the RAID system which could cause data loss. Your Linode needs to be migrated to a new server immediately to avoid extended downtime or data loss. Due to the severity of this issue, we decided that an emergency migration is more prudent than scheduling maintenance.

The maintenance will start on Saturday, August 5 at 9PM EDT (Sunday, August 6 at 1AM UTC).

When your Linode migrates, it will be cleanly shutdown, migrated to its new hardware, and then returned to its last state (running or powered off). You can monitor your position in the migration queue from your Linode’s Dashboard. All of your data and any IP addresses assigned to your Linode will be migrated as well. The migration process will move your Linode’s disk images at the rate of 3-5 GB of data per minute.

Update: Completed without problems.

Server upgrade, data breach on old server - change your password!

Tim Starling from Wikimedia has kindly helped upgrade RationalWiki to MediaWiki 1.27. This will be going live shortly. This should bring us many functionality and security improvements.

In the process, Tim discovered that, in February 2017, the RationalWiki site was breached and the site's user table was downloaded. The user table contained:

• Password hashes. "Because the hash used by MW before version 1.24 is cheap to calculate on a GPU, you can invert even moderately good passwords hashes, like 8 random alphanumeric characters."
• Email address associated with an account, which could be associated with a password hash.

Users should change their password, and change it anywhere else they've used that password.

Tim thinks the breach was a drive-by opportunist, rather than someone targeting RW specifically.

EvolutionWiki is dead, long live RationalWiki

EvolutionWiki was a wiki to collect skeptical information to fight creationism. When it shut down, RationalWiki took it on to port the useful stuff over. Tim Starling noted it was an ancient unmaintained security hazard, and we'd ported most of it to the main RationalWiki anyway, so David just killed it - it now redirects to the corresponding page name on RationalWiki. See discussion.

Server shuffling and upgrades

With the kind volunteer assistance of Tim Starling (from Wikimedia), we're working on shuffling the RationalWiki servers around and upgrading at last.

First thing in the programme: around 1400 UTC today (3pm BST, 10am EDT, 7am PDT), David is about to repoint DNS at apache1, and switch off and delete the Squids and the load balancer. Users should notice no effect, but we'll be keeping watch.

Next up: set up new servers with up-to-date software, SSL termination at last ...

Update: All done. You should have seen no effect whatsoever. More to come!

MySQL problems back

This MySQL bug has shown up again. I am at the day job right now but will be attending to this from about 1900 UTC.

Update: All appears well, it was MySQL having a conniption. Now replacing it with MariaDB ...

MySQL database problems, work in progress

The MySQL database has been showing errors for the past few days. The error.log suggests this indicates a corrupt table, and to dump the data and restore from fresh. David is currently trying to get a good dump, per this manual page which is only slightly more than a little trepidatious. In the meantime, the wiki is down to avoid making the problem worse. If all else fails we have a VM snapshot from this morning (though that still has the corruption problem and will need recovery). Live updates on the RationalWiki Twitter.

Update: Recovered now. A dizzying array of backup mechanisms will be put into place ... tomorrow.

Wikimedia Commons images are working again

RationalWiki uses images from Wikimedia Commons where available, via the InstantCommons mechanism. This broke last week because Commons' API went HTTPS-only. This is fixed on RW now, though you will see HTTP pages with HTTPS images.

(yes, we still need to get around to HTTPS on the wiki itself)

Downtime Fri 03 Jul 2015, ~12:30-13:30 UTC

After much puzzling around MySQL, it seems apache2 was hammering the living my-goodness out of the MySQL instance on apache1. There is no obvious reason for this to have occurred ... but I disabled Apache on apache2 on the assumption that flying on one engine beats crashing on two. Investigation proceeds, further hiccups may occur.

Update: The answer: MySQL on apache1 saw apache2 as coming from its internal IP, so connections from it hung when it tried a reverse DNS lookup on this IP. Simple fix: skip-name-resolve in my.cnf.

Downtime weekend of 4/19/2014

UPDATE 04/20/14-11:40AM EDT

We have started the update for the database server so will be offline for at least several hours. I will be monitoring the situation and get everything back up as soon as possible. 

UPDATE 04/20/14-2:00AM EDT

First round of updates went smoothly with no downtime or issues. Tomorrow's update will cause downtime though. Probably 3-4 hours.

ORIGINAL POST:

Linode, our hosting service, has upgraded their services and in order to take advantage of the upgrades we need to power down our machines. This first phase of this upgrade will occur on Saturday 4/19/2014, and will involve upgrading our frontend and apache clone. This should not require taking the site offline but could potential cause strange behavior or decreased performance.

The second phase will occur at 12:00pm EDT Sunday, 4/20/2014 and will require shutting down our database server. This will cause the site to go offline. We estimate that this should take 3-4 hours. 

For more information on the upgrades see Linodes blog post.

Connectivity issues at our hosting facility

The Newark hosting facility where our servers are located is experiencing connectivity issues. This does affect the RationalWiki websites, status updates are available here.

Incoming! Rapid server reconfiguration underway ...

RationalWiki just got sued by Kent Hovind. The lawsuit is a comedy classic and will brighten your Friday afternoon. (Saloon Bar discussion.)

We're not worried about the suit — the worry is that we're about to get a zillion well-wishers hammering the server. Trent is frantically rejigging our setup as we write this, pressing a spare box into service. More news as it comes, and if the site entirely vanishes please try again five minutes later ...

Update, next day: We quickly spun up a 4GB Linode to just put Apache on (MediaWiki scales horizontally really well). It appears that Apache was taking lots of CPU, not MySQL! So we've split Apache for the main site between the two boxes (the old 8GB and the new 4GB) and the setup is ticking along nicely. Trent may have some nice graphs later.

The wiki is getting hammered.

Haven't checked the Squid logs as to what it is —I suspect fallout from the Nye/Ham debate. But we're getting a buttload of requests, a lot of Apache processes, load around 50-55, MySQL using 70% of CPU after a restart ... we're not actually running out of memory, the wiki's just legitimately doing a lot of work. So page loads are often working, but taking up to a minute, and there's lots of timeouts. David's keeping an eye on stuff.

Edit: It doesn't look like any particular article. Our creationism stuff in general is getting lots of hits. While youre waiting, here's today's Google cache of "How come there are still monkeys?"

Edit 2: Still very highly loaded, but is serving up pages now, if very slowly.

Apache zombie process problems.

The Apache server is being a dick. It keeps dying with unkillable zombie processes; these don't serve data, but they do keep their hold on port 80. The only workaround is to reboot the server. This is, of course, ridiculous. I've put a ticket in with Linode. Anyone got ideas? It's just the standard Ubuntu apache2 package.

Update: Apache was causing kernel oopses. (Things that make you go "wtf.") Linode suggest using kernel 3.9-linode instead of 2.6-linode (the default for Ubuntu 10.04, which our box runs on). We'll see how that goes next reboot. Might even fix the white-screening blog too.

Wiki slow, we're being hammered.

The wiki has been really slow all day. The Apache server and the Squids appear quite happy with life - but the load balancer is reporting huge traffic spikes. More information as I have it.

(The blogs are unaffected, as they're served directly from the Apache server.)

Update: Here's the load balancer connections per second. (The gaps are when logging is killed to increase throughput or something.) Note the bit where we go from a few tens of queries a second up to 800 a second.

Site is doing its yoyo impression again.

I'm not sure what's going on (and I'm currently without broadband, so I don't have reliable internet time to diagnose the problem), but the Apache server is occasionally going into conniptions and needing a reboot. (It's in one as I speak.) We are aware of the problem ... fwiw.

State of the blog: recovering.

And the blog returns from its holiday at the seaside. The blog is now happily resident on the main RationalWiki server. Just waiting for the DNS to resolve everywhere. I also need to fix the front page images (missing because of the way I did the import).
 
The next holiday at the seaside will probably be in a week or two, when I’m busy moving house. But I’m sure our contributors will write in great volume to take up the slack. Just as soon as I get around to recreating their accounts on this instance.

We also have an insanely dull front page, just waiting for a burgeoning blog network full of interesting people to spontaneously manifest from the quantum ether.

State of the blog: addled.

The site the RationalWiki Blog is presently living on is fine for what it is, but what it is includes not being up to our needs, hence the blog's party-stopping yoyo impression for much of today. We're finally getting around to doing the sensible thing and setting it up on the main RationalWiki server (which just got its memory doubled, so there's actually room for stuff). Please excuse hiccups in the meantime.

Site slowness.

The site is frequently running ridiculously slowly. At present I don't really know why. Sometimes it's obvious (load average 30, Apache/PHP compiling lots of slow queries running), sometimes it isn't (when it was slow this afternoon). The system still has plenty of memory free. The squids don't show an unusual rate of hits on the site. Further updates at RationalWiki:Technical support. At least we can all be puzzled in a fully informed manner.

Short downtime for upgrade some time during 10-11 April 2013.

RationalWiki gets an upgrade! Per the details here, Linode has doubled everyone's RAM for free. Trent will be putting the various machines into upgrade queues today. He will stagger the upgrades starting with the Squids to limit downtime. However, when the backend is upgraded the site will not be available.

(We are very happy customers of Linode and can heartily recommend their service. This is a referer link that gives RWF a few pennies.)