Saturday, June 17, 2017

EvolutionWiki is dead, long live RationalWiki

EvolutionWiki was a wiki to collect skeptical information to fight creationism. When it shut down, RationalWiki took it on to port the useful stuff over. Tim Starling noted it was an ancient unmaintained security hazard, and we'd ported most of it to the main RationalWiki anyway, so David just killed it - it now redirects to the corresponding page name on RationalWiki. See discussion.

Server shuffling and upgrades

With the kind volunteer assistance of Tim Starling (from Wikimedia), we're working on shuffling the RationalWiki servers around and upgrading at last.

First thing in the programme: around 1400 UTC today (3pm BST, 10am EDT, 7am PDT), David is about to repoint DNS at apache1, and switch off and delete the Squids and the load balancer. Users should notice no effect, but we'll be keeping watch.

Next up: set up new servers with up-to-date software, SSL termination at last ...

Update: All done. You should have seen no effect whatsoever. More to come!

Thursday, April 7, 2016

MySQL problems back

This MySQL bug has shown up again. I am at the day job right now but will be attending to this from about 1900 UTC.

Update: All appears well, it was MySQL having a conniption. Now replacing it with MariaDB ...

Saturday, March 5, 2016

MySQL database problems, work in progress

The MySQL database has been showing errors for the past few days. The error.log suggests this indicates a corrupt table, and to dump the data and restore from fresh. David is currently trying to get a good dump, per this manual page which is only slightly more than a little trepidatious. In the meantime, the wiki is down to avoid making the problem worse. If all else fails we have a VM snapshot from this morning (though that still has the corruption problem and will need recovery). Live updates on the RationalWiki Twitter.

Update: Recovered now. A dizzying array of backup mechanisms will be put into place ... tomorrow.

Tuesday, November 10, 2015

Wikimedia Commons images are working again

RationalWiki uses images from Wikimedia Commons where available, via the InstantCommons mechanism. This broke last week because Commons' API went HTTPS-only. This is fixed on RW now, though you will see HTTP pages with HTTPS images.

(yes, we still need to get around to HTTPS on the wiki itself)

Friday, July 3, 2015

Downtime Fri 03 Jul 2015, ~12:30-13:30 UTC

After much puzzling around MySQL, it seems apache2 was hammering the living my-goodness out of the MySQL instance on apache1. There is no obvious reason for this to have occurred ... but I disabled Apache on apache2 on the assumption that flying on one engine beats crashing on two. Investigation proceeds, further hiccups may occur.

Update: The answer: MySQL on apache1 saw apache2 as coming from its internal IP, so connections from it hung when it tried a reverse DNS lookup on this IP. Simple fix: skip-name-resolve in my.cnf.

Friday, April 18, 2014

Downtime weekend of 4/19/2014

UPDATE 04/20/14-11:40AM EDT

We have started the update for the database server so will be offline for at least several hours. I will be monitoring the situation and get everything back up as soon as possible. 

UPDATE 04/20/14-2:00AM EDT

First round of updates went smoothly with no downtime or issues. Tomorrow's update will cause downtime though. Probably 3-4 hours.

ORIGINAL POST:

Linode, our hosting service, has upgraded their services and in order to take advantage of the upgrades we need to power down our machines. This first phase of this upgrade will occur on Saturday 4/19/2014, and will involve upgrading our frontend and apache clone. This should not require taking the site offline but could potential cause strange behavior or decreased performance.

The second phase will occur at 12:00pm EDT Sunday, 4/20/2014 and will require shutting down our database server. This will cause the site to go offline. We estimate that this should take 3-4 hours. 

For more information on the upgrades see Linodes blog post.