Downtime for upgrade: Saturday 6 April 2019, 12:00 UTC-14:00 UTC

We get a free upgrade from Linode! We just have to press the button and wait 90 minutes for them to migrate the server.

I'll be pressing the button around 12:00 UTC on Saturday 6 April. Go outside or something while it's happening.

Update, 13:17 UTC: aaaand we're back!

Emergency maintenance, Sun 06 Aug 2017, 0100 UTC on

Linode (our hosting company) has discovered hardware problems, with action to start at 0100 UTC tomorrow morning:

We continuously monitor the health of our equipment, and we've been alerted to an issue with the physical hardware on which your Linode resides. Specifically, there is a hardware issue with the RAID system which could cause data loss. Your Linode needs to be migrated to a new server immediately to avoid extended downtime or data loss. Due to the severity of this issue, we decided that an emergency migration is more prudent than scheduling maintenance.

The maintenance will start on Saturday, August 5 at 9PM EDT (Sunday, August 6 at 1AM UTC).

When your Linode migrates, it will be cleanly shutdown, migrated to its new hardware, and then returned to its last state (running or powered off). You can monitor your position in the migration queue from your Linode’s Dashboard. All of your data and any IP addresses assigned to your Linode will be migrated as well. The migration process will move your Linode’s disk images at the rate of 3-5 GB of data per minute.

Update: Completed without problems.

Server upgrade, data breach on old server - change your password!

Tim Starling from Wikimedia has kindly helped upgrade RationalWiki to MediaWiki 1.27. This will be going live shortly. This should bring us many functionality and security improvements.

In the process, Tim discovered that, in February 2017, the RationalWiki site was breached and the site's user table was downloaded. The user table contained:

• Password hashes. "Because the hash used by MW before version 1.24 is cheap to calculate on a GPU, you can invert even moderately good passwords hashes, like 8 random alphanumeric characters."
• Email address associated with an account, which could be associated with a password hash.

Users should change their password, and change it anywhere else they've used that password.

Tim thinks the breach was a drive-by opportunist, rather than someone targeting RW specifically.

EvolutionWiki is dead, long live RationalWiki

EvolutionWiki was a wiki to collect skeptical information to fight creationism. When it shut down, RationalWiki took it on to port the useful stuff over. Tim Starling noted it was an ancient unmaintained security hazard, and we'd ported most of it to the main RationalWiki anyway, so David just killed it - it now redirects to the corresponding page name on RationalWiki. See discussion.

Server shuffling and upgrades

With the kind volunteer assistance of Tim Starling (from Wikimedia), we're working on shuffling the RationalWiki servers around and upgrading at last.

First thing in the programme: around 1400 UTC today (3pm BST, 10am EDT, 7am PDT), David is about to repoint DNS at apache1, and switch off and delete the Squids and the load balancer. Users should notice no effect, but we'll be keeping watch.

Next up: set up new servers with up-to-date software, SSL termination at last ...

Update: All done. You should have seen no effect whatsoever. More to come!

MySQL problems back

This MySQL bug has shown up again. I am at the day job right now but will be attending to this from about 1900 UTC.

Update: All appears well, it was MySQL having a conniption. Now replacing it with MariaDB ...

MySQL database problems, work in progress

The MySQL database has been showing errors for the past few days. The error.log suggests this indicates a corrupt table, and to dump the data and restore from fresh. David is currently trying to get a good dump, per this manual page which is only slightly more than a little trepidatious. In the meantime, the wiki is down to avoid making the problem worse. If all else fails we have a VM snapshot from this morning (though that still has the corruption problem and will need recovery). Live updates on the RationalWiki Twitter.

Update: Recovered now. A dizzying array of backup mechanisms will be put into place ... tomorrow.