This MySQL bug has shown up again. I am at the day job right now but will be attending to this from about 1900 UTC.
Update: All appears well, it was MySQL having a conniption. Now replacing it with MariaDB ...
Update: Recovered now. A dizzying array of backup mechanisms will be put into place ... tomorrow.
RationalWiki uses images from Wikimedia Commons where available, via the InstantCommons mechanism. This broke last week because Commons' API went HTTPS-only. This is fixed on RW now, though you will see HTTP pages with HTTPS images.
(yes, we still need to get around to HTTPS on the wiki itself)
After much puzzling around MySQL, it seems apache2 was hammering the living my-goodness out of the MySQL instance on apache1. There is no obvious reason for this to have occurred ... but I disabled Apache on apache2 on the assumption that flying on one engine beats crashing on two. Investigation proceeds, further hiccups may occur.
Update: The answer: MySQL on apache1 saw apache2 as coming from its internal IP, so connections from it hung when it tried a reverse DNS lookup on this IP. Simple fix: skip-name-resolve in my.cnf.
We're not worried about the suit — the worry is that we're about to get a zillion well-wishers hammering the server. Trent is frantically rejigging our setup as we write this, pressing a spare box into service. More news as it comes, and if the site entirely vanishes please try again five minutes later ...
Update, next day: We quickly spun up a 4GB Linode to just put Apache on (MediaWiki scales horizontally really well). It appears that Apache was taking lots of CPU, not MySQL! So we've split Apache for the main site between the two boxes (the old 8GB and the new 4GB) and the setup is ticking along nicely. Trent may have some nice graphs later.