RationalWiki

Traffic spike post-mortem

2011-04-12T19:44:00.003-04:00

As many of you are already aware there was a huge traffic spike to RW over the weekend. One of our tumble weed articles (Scientific evidence of evolution being a hoax) hit the big time, again actually.

At its peak on Sunday we had made the front page of Reddit, all over Facebook and several large blogs. Over 100,000 people accessed our site during this time and the site became almost impossible to use.

I understand the frustration that this can potentially cause, and several people initiated discussions about what to do to allow RW to withstand these kinds of spikes.

My response to this is that it doesn't make sense to develop our infrastructure using this weekends traffic loads as our upper bound tolerance level. This is the single largest traffic spike in RW history, and traffic spikes big enough to seriously impact the performance of the site happen only a few times a year. The infrastructure we would need to be able to keep RW working without impact under that kind of traffic load would likely increase our costs 10 fold.

Our existing infrastructure supports our average traffic load, and can support surges several times what we get as baseline. When are greatest limiting resource is financial we have to be smart about how we spend our money. It is better to struggle a few times a year under extreme traffic load, then to go broke in a month and have to shut down operations.

With that said, I am constantly monitoring server performance, and the increase in baseline traffic that we get. Our baseline traffic is continuing to grow at a significant rate, and the rate its growing has been constant for over a year. As the baseline demands increase, I have moved and altered the server infrastructure to support it within the financial limits we have.

The hope is that as our requirements increase, so does the pool of users that are willing to help us out with a few bucks along the way. Thus allowing us to expand our infrastructure as needed. So far this is proving to work out well. We are in the middle of a current fund raiser on RationalWiki that is on track to be our most successful yet. If you haven't had a chance to toss a few dollars our way, now would be a great time!

Ultimately, I think we should celebrate our growing popularity, and remember that growing pains are signs of just that: growth!

Site issues

2011-03-29T12:01:00.003-04:00

I am aware that there are issues and am actively working on finding the problem and a solution. That's all I have for now.

UPDATE:

I have installed some tools to help keep tabs on the server since I can't really do it 24/7 and detecting patterns and glitches is difficult. Hopefully this will allow me to better respond to problems. For now though I will try and keep things running as smoothly as possible till everything blows over.

Worry free vacation?

2010-12-16T11:16:00.002-05:00

I am leaving, and this time if a tree falls on the power lines to my house or my roommates don't turn off the stove and burn everything to the ground RW will carry on! No need to spend the holidays invading other poor wikis.

Tentatively open for bussiness

2010-11-14T12:33:00.001-05:00

The site is back up and open at: rationalbeta.com let us know of any problems.

The normal domain will point to the new server eventually.

Since I know you guys are all at church

2010-11-14T08:56:00.003-05:00

We are starting the server transfer, RationalWiki is now locked down. We need to sync the file system and the database to the new server and then do some magic and then see if it all worked out.

It will take a few hours to a few days depending on the internet gods till rationalwiki.org points to the new server.

However, rationalbeta.com should be pointing to the new server for everyone and the ip address 173.255.233.133 should also default to RationalWiki. I will announce everywhere when the server move is complete and normal operations can resume at the new site, and I hear tell that there is also going to be some celebrating at the old site.

God's impotent rage!

2010-10-29T16:17:00.002-04:00

Once again God tried to smite our site, this time by blowing down a tree limb on the power lines to my neighborhood. But he forgot about the utility people who fixed.

But apparently my modem didn't boot back up correctly and that's why the site was down.

Maybe its time to start thinking about commercial hosting again. I am making more money a month than I was 3 years ago......

Yep its down

2010-10-29T15:52:00.001-04:00

I am currently in the lab, and so don't know anymore than you guys do. I will try and get home and check on it as soon as I can. If its internet related I might not be able to write back till its up, but if its hardware or software related I will try and get back to you guys with an estimated time till fix.

Damn the man!

2010-09-07T12:49:00.001-04:00

Well we should be back up for everyone, but there are...issues...I should have a work around in place for now as I contemplate what to do next. But there shouldn't be any obvious effect on RW anytime soon.

Static IP installation

2010-09-06T23:40:00.001-04:00

As alluded to earlier we need to get a static IP address installed. They are suppose to come sometime tomorrow and do it. It involves new network hardware that will have to be integrated into the existing setup, and a new IP address which will then have to be propagated from the DNS.

Expect downtime (September 7th, 2010). Details posted here as needed.

Coming back up

2010-08-26T12:29:00.005-04:00

Obviously there has been some issues. I had a problem with the network on this end that was cutting off my internet access. This, of course, made it impossible to connect to the server but also limited my ability to post to let you know what was up.

I think I have got a temporary solution worked out so that things should be coming back on line shortly.

UPDATE: The network can see the internet again (obviously I am posting) but there are issues with our reserved IP address. It seems that my ISP has switched to true static rather than the fake static they had been using. I can get a static free of charge with current internet package but it takes 8-10 days. In the mean time our IP has changed and the new IP has to be propagated.

Short answer: RW will has a new IP address and I need to change the DNS. Your access to RW will return whenever your DNS updates which should be a few hours but could be longer.

UPDATE 2: DNS is changed and should be propagating. It is up for me at least and should be fore you soon.

The pipe is leaking somewhere

2010-08-20T15:03:00.001-04:00

Somewhere along the intertube to the server there is a clog, I am not sure yet if the problem is technical, on my side or my ISP side, or if its a traffic influx. More to come, but there might be some slow down on RW.

Backup

2010-08-02T16:12:00.003-04:00

I am in the process of reworking the backup script and its timing. I think I am going to be dividing "jobs" up a bit to spread the downtime around a little bit. Nothing will be happening before Midnight EST, but the timing maybe changing over the next few weeks as I tweak things out to try and decrease the footprint of the backup on the site itself.

Traffic spike redux

2010-05-19T15:51:00.003-04:00

Shouldn't be nearly as bad as last time traced to:

Pharyngula

Power outage

2010-05-18T19:15:00.000-04:00

Power outage took out the server, things are back on now but looks like the MYSQL tables need to be checked. Site should be back on line shortly.

Upgrade, phase 2

2010-05-08T23:58:00.000-04:00

I have officially started phase 2b of the upgrade procedure. Hence the site will be down.

Proof of concept fail

2010-05-05T20:42:00.001-04:00

So my proof of concept test failed for the "easy" setup version of my plan. But the pieces are all ready in place so just pushing forward with the "harder" version. I will be as quick as I can. Sorry for lack of real notice.

UPDATE(11:53pm 5/6):
You reach a point where you are far enough along you might as well push to the finish. Lets just get this all done while its all broken. Problems abound, but I will fix them eventually. It will be up before I go to sleep.

I give up!

2010-04-29T14:40:00.002-04:00

There is nothing else I can do right at this moment to improve usability of the site.

The Internet's attention span can not last much longer....I hope.

There are potentially several bottlenecks where things are hanging, some can be addressed with some hardware upgrades, others maybe more complicated to figure out.

I don't think we are at a do or die stage for getting this figured out, as the vast majority of this is "drive-by" traffic that should probably lesson considerably in a matter of days.

Tumble weeds

2010-04-26T01:17:00.002-04:00

Well, people really love those tumble weeds. We are still getting hit with a lot of incoming traffic to that page. It started with some facebook posts, and now its spreading to some of the other social bookmarking type sites.

The internet's attention span shouldn't last much longer and RW will be back to normal. But we are likely to experience some slow downs and other issues (particularly with editing) till it passes.

Time to start thinking about expanding load capacity maybe?

Site access issues

2010-02-09T13:29:00.003-05:00

There are some issues that are effecting site access. At this point all I have to offer is that I am aware of the issue, have a few "band-aid fixes" I can use, and will try and figure out whats going on as soon as I can.

RationalWiki: Year in review

2010-01-26T18:12:00.013-05:00

Lots has happened since the start of 2009, it was January 4th that RationalWiki opened up on our own dedicated server residing in the corner of my room and while there have been ups and downs I think over all it has worked out well for the site. Plenty happened on the site but this post is not about that. Instead I wanted to take a look at a years worth of fun statistics that very few people usually get to look at.

Overview:

Lets start with the overview then shall we:

So over the course of 2009 we had just over 825,000 unique visitors. This is measured by unique IP addresses so take it with a grain of salt. It means essentially that 825,000 different IP addresses have requested data from our site.

Close to 1.6 million "visits" over this time period as well. Visits is hard to clearly define, but its an attempted measure of someone who comes to the site, stays on the site, then leaves. That counts as a single visit. So the same person will usually have multiple visits to the site a day.

Pages and hits are tough to really do much with on a wiki since by the nature of the software they have a tendency to be inflated.

Bandwidth, you guys sucked 584 gigabytes of data through my internet line. That's the equivalent of:

600 standard music CDs
124 feature length films
All the books and journals in the library I am writing this in now

Plenty of other fun measurements, but its a lot of data. All the text on the wiki right now takes up about 12 gigabytes of diskspace. That's a lot of text.

Matters of time:

A graph of usage based on the days of the week and then hours of the day:

What this tells us is a few things, our most active days are Monday-Thursday, we drop off on Friday and a lot on Saturday, increasing slightly on Sunday.

The vast majority of our usage is between about 7am-5pm EST. While our active user base is fairly geographically distributed, the majority of our traffic still comes from US based sources. Combining the days and the hours that we seek peek popularity I think RationalWiki is popular for people who are at work.

Where?

This is some information about "where" are visitors are coming from. There are a lot of problems with it so don't put much stock in it. This work by trying to find the top level domain for who a user is, if you are an ip address you are an unknown. But a lot of people you can actually trace back to the TLD of their ISP. The .com is mostly US based traffic, from there you can see we pull a lot of traffic from Hungary, more on that latter. Also the UK and Australia domains are up there as well.

On what?

Here is a question that pops up every now and then on a wiki, what are people using to see our site?

As you can see the windows operating system is the dominate OS of choice. However, the dominate Microsoft browser is actually less than a third of our traffic. Firefox dominates that category. Good for you guys! Now if I could just convince you all to install Linux....

Referrals:

This data is a little harder to visual, so take my word for it. Our largest referral sites for the year have been reddit, fark and something awful forums. Reddit accounts for something like 65,000 refer hits, fark closer to 50,000, and something awful about half that again.

However we did have a few interesting referrals. The google translator (http://translate.googleusercontent.com) is rapidly becoming a major source of traffic. A few random blogs in various languages have also been linking to us. For example, a few popular gaming blogs in Hungary linked to us on Poe's Law while discussing the "anti-spore game website."

Is it time for hu.rationalwiki.com? Probably not.

As for search engines themselves:

As you can see google is the dominate one here. Followed by stumbleupon which is more "social bookmarking" than search engines, but we can pull in a lot of traffic from a popular book mark on stumble upon.

But we all probably need google was our main search engine referrer, the real question, and often the more perplexing of all, is what people search for when they find us?

Poe's Law and gerbling:

So let us start with the big ones:

So people found RationalWiki searching for 150,000 different things. About 65,000 searches were people looking directly for us. Another 10,000 were people looking for conservapedia.

Of course the big one, by far, is Poe's Law. It has sucked in a lot of search engine traffic. When you combine the blog links and the searches it accounts for a large number of the people coming to our site. It is a fun success story, we need to find more "poe's law" articles to write people!

Gish Gallop and our Esther Hicks article get an honorable mention here as well.

Really though the vast majority of searches are for a random things. Here is a sample:

historical contingency
project blue beam wiki
gerbiling
venomfangx
golgi bodies
trilateral commission logo
fun ways to master bait
homotoxicology
st. isaiah - archangel of divine mercy
maltheism
infant damnation
bridgewater triangle
should the government provide healthcare
world s largest rat
mythical goats
recipe for donair sauce

Some stats updates

2010-01-18T02:09:00.003-05:00

I haven't done much in the way of a stats update in a while. I think the last one I really talked about was back in October. While are total numbers in visitors has stayed fairly constant the actual shape of the traffic has changed some what.

Previous months our traffic has been shaped by major events, where we get lots of spikes and valleys. Lately we have been getting less spikes (with a few exceptions) but our base traffic level has increased substantially. So we are getting 5k-6k "visits" a day now, where as even a few months ago we were at 3k-4k with spikes of 8k-9k.

Our incoming link data is also changing a bit. Previously, it was dominated by one or two major links from places like reddit, or something awful. Last few months its been spread out across a whole lot of mid-range blogs, covering a wide range of topics. Poe's Law still nets us the most links, but things like Gish Gallop, and other more specific articles are starting to get a lot of "passive linking" as well.

I think this change in traffic is also reflective of changes we are seeing on the site itself. We are now pushing 170+ active editors, recent changes no longer allows me to see everything that happened in a day, and we are starting to feel those growing pains as we work out management issues on the site.

Short version: more people are coming to our website every month, and the traffic pattern is becoming closer and closer to what I consider our ideal, of lots of incoming links in the style of "go here for more information about this topic." This is good for our site, good for our mission, but is also causing growing pains. It will be interesting to see how all this plays out over the next year.

Site down

2010-01-14T11:32:00.004-05:00

I am aware of the issue and working on bringing it up as we speak. It is a software issue which means this shouldn't take too long. Updates to follow.

Update: Still working on it. My initial optimism is now met with existential pessimism. Any apache experts want to chime in on this one?

Update 2: Alas, once again my existentialism dies in its infancy. The site is back up, go forth, and be fruitful or something.

Colbert Nation

2009-12-09T00:08:00.001-05:00

The server is under tremendous load tonight, most likely from our top spots for relevant google searches after Andrew Schlafly's appearance the Colbert show.

I will do what I can behind the scenes to maximize our ability to weather the storm, but it is likely to be rough for a day or so. I imagine the initial rush should be only a few hours.

Techinical issues...

2009-11-26T02:22:00.005-05:00

If you have tried to access RW in the last hour or so you may have noticed we are having some technical issues.

Working on it while I type, and the good news is that the server was able to recover from a crash to a state that remote access was available. This is important because it means our fault protection appears to be working allowing for remote recovery from issues.

Hoping to have everything back and working with in the hour.

Update: Site recovery went fine, but rerunning back up since it got jacked up, so looking at another 20 minutes or so till the site is fully accessible again.

UPDATE: Okay site should be back up and running as normal. Its well past my bedtime.....have to teach tomorrow morning. OH well. Leave a message on my talk page or on here if you have problems with the site or you can contact Nx as he should be able to handle anything. He is still here right?

October stats

2009-11-01T00:12:00.003-04:00

October was the most active month ever for RW. There were multiple events that helped spur us on to nearly 100,000 unique IP address visits. We also appear to have recovered from our slump due to the RW outage in August/September.

Below is the daily traffic record. We had three major events that helped propel our traffic. I will talk about those below.

Point A was of course the The Conservative Bible Project.

Point B was a couple retrospect posts about the Lenski affair. Such as this piece from slashdot.

Point C was the telegraph article on internet laws.

Traffic spike

2009-10-06T00:21:00.002-04:00

Due to the coverage of The Conservative Bible Project RW is seeing a predictable jump in traffic. Its still going hot and in order to keep serving up are brand of the good news I am holding off on the back up for tonight.

Don't break anything.

An act of "nature"

2009-10-03T17:34:00.003-04:00

Sorry folks, based on the blinking clock above my computer it appears the power went out to my house. All the various fault protection equipment kicked in as appropriate but I can't sustain the server for longer than about half an hour with no power. Based on site monitoring it appears that it was inaccessible for only a few minutes at most.

Fault protection take 2

2009-09-25T14:34:00.003-04:00

I have received the additional hardware I needed to get the fault protection working...I think. I will be setting that up and running some test on it today. It shouldn't effect RationalWiki at all as I can do my testing further down the network chain. If all goes well I will need to restart the server and that's about it. I will drop an intercom on RW when/if I do that.

Update: Okay, fault protection seems to be working well. I have implemented it live on the server now. I will continue to monitor everything and adjust as needed. But all appears well for now.

So just how bad has it been?

2009-09-23T16:20:00.003-04:00

The recent downtime issues annoy me at least (probably more) than every user of RationalWiki. As always, I strive to offer the best service as far as performance, availability, and protection against catastrophe that I can. But I am not perfect, and I think it is important to keep in mind that this is hobby, and my "real life" as a graduate student is as overworked and under paid as most cliches depict it.

All that said and done, just how bad has it really been? After the great crash in August I started working with a service to help monitor RationalWiki's up time and server performance. With close to three weeks worth of data we can paint a picture for how bad things have been.

If you take a look at this analysis you can see that RW has actually been up 95 percent of the time. That is not bad all things considered. Take into account a few points: first that a good chunk of our downtime is packed together so it is mostly caused by 1 or 2 disasters that caused prolonged downtime, and second that our nightly backups cause time out errors for about 20-30 minutes. If you remove the few major disasters our uptime averages just over 99 percent a day, and without the backups you are looking at 100 percent coverage most days.

The key then is disaster recovery. To be able to quickly handle issues that cause long protracted downtime. Most of these are easily handled if I am awake and with in walking distance of the server. The issues today with the cable going down are very rare. So we are left with one major issues: server cop-outs that prevent remote log in and shutdown the site that occur when I am either asleep or traveling.

I am actively working on a solution that I think will greatly increase the servers ability to auto-recover from failure, and to expand the options for remote administration in the event of catastrophic failure when I am not present (ala what happened in August).

A lot of this is trial-by-error and learning as I go. I have never done a project like this before. All we can do is learn from our mistakes, and move forward with the goal of doing the best we can. That said what we do have is pretty good I think.

And we are back....

2009-09-23T16:06:00.002-04:00

What appears to have happened is my next door neighbors got a cable hookup, and God put it into the mind of the cable technician to accidentally disconnect my cable while doing the hook up.

Where God failed in his plan was in prevent another technician from showing up and fixing the problem. An even larger failure was that he allowed the technician to show up hours early! How often does that happen? I think we know who to thank for that!

On a more serious note, in my comment on the last post I discussed the monthly cost of running RW. The point was not to ask for more money, the cost stems form the dedicated IP address and extra bandwidth that RW requires. It is an extra $50 a month above what I would pay anyway. I get about $20 a month in donations from people that give every month. That means my per month out of pocket expense is $30 a month. I can totally handle that. The occasional need for new hardware that I can't afford is usually met almost immediately by a small donation drive. Every now and then I get an extra donation that I keep on the back burner for emergencies.

The RW accounts have about $80 sitting in them right now for emergency purchases. I like to keep that amount around $100-$150 to make sure I can get almost anything we need.

All of this is to say that at the moment the financial cost of RW is not really a burden on anyone. That was the point to moving to a privately hosted site.

If we were to try and move to a commercially hosted site the financial burden increases multiple fold for me and those people who are able and willing to donate. There is also substantial risk that if we fail to get enough cash we could lose the site.

So we put up with this less than perfect up time because it means that RW is under no immediate risk of permanent shutdown and is not a going to bankrupt anyone in the proccess.

When it rains...

2009-09-23T12:51:00.000-04:00

So God's will has struck again, the site is down but this time it has nothing to do with the server. My internet connection seems to have crapped out. Because I pay $200 a month for it though one of the perks is "emergency service" so I have a guy coming in sometime beteween 5pm-8pm EST to take a look at it and hopefully get everything back and running.

No sleep for the server admin

2009-09-22T16:24:00.002-04:00

So I was up till 4 am trying to get the fault protection working. It appeared to be working beautifully so I went to bed. Only to be awoken 3 hours later with it going nuts. I have removed it from the system for now. I think I need another piece of hardware to get everything working together the way I want. So I am putting it all on hold for now. I will order the new hardware today so it could be upwards of a week before I go at this again.

My goal is to have it all in place before my trip to Chicago.

And now I think I am getting a cold. I blame the stress and lack of sleep damn it.

Okay I lied

2009-09-22T03:40:00.002-04:00

As is probably obvious I lied in the last updated when an idea for a solution to my problem came to my mind.

So I think I have everything setup. I am going to avoid going into specifics for security reasons but we now have much greater remote administration abilities that are no longer dependent on the server being online. I have also setup a range of automated monitoring software and utilities that will aid in both keeping track of site up time and doing some automated tasks that should allow the server to recover from all but the most serious of crashes automatically.

With that I am going to bed.

Extended maintenance

2009-09-21T16:45:00.002-04:00

The site is going down this afternoon for extended maintenance. Running some tests, changing some options and install some new hardware. All designed to try and help deal with some of the recent outages. Running the backup first, then I will get started.

Update: Screw it I am done for tonight. Got about half of what I wanted figured out. Luckily the last half lets me keep RW up most of the time I am working. I will have to come back and keep working on this probably tomorrow which means don't freak if there is intermittent downtime for a minute or so every now and then for the next day or so.

Repairing a table in the database

2009-09-21T11:01:00.001-04:00

Things will be locked up for a few minutes while the repair is run. I am aware of the situation and working on it, and hope to have things back up shortly.

UPDATE: Repair is done site is back online let me know if there are further issues.

Server crash post-mortem

2009-09-13T18:47:00.002-04:00

Time for the official post-mortem of what happened as far as the server crash goes. The official cause of the crash shall be listed as a failing power supply unit.

About a month ago the power supply for the server went completely dead. In order to get the server back up and running as quickly as possible I swapped in a spare unit I had from an older computer. It did the job beautifully. Seeing as how everything appeared to be working fine and there were no substantial problems I didn't replace it with a new unit.

About 4 days before I left for my trip back home the server shut itself down. When I booted it back up I had some problems with the MYSQL server and so chalked the problem up to that. Then I left and went away. We all know what happened next. When I got back to the server I found that it was in the same disabled state as the first crash. I got it back up and decided to watch and see what it would do. A day later the same thing happened.

So I ran some tests on the power supply and it was providing irregular power on the 12v rail, my guess is that was probably leading to a temperature triggered shutdown. Anyway, regardless, 2 days ago I purchased a high-end power supply unit and swapped it in. Everything seems to be running fine now.

Thanks to the donations everyone at RW gave or have promised to give, I have gone ahead and upgraded some of the networking hardware that was worrying me as well. I am also working on getting some hardware to allow for remote management of the server even if it is unresponsive, as well as server resets automatically if it becomes unresponsive.

So the whole thing is my fault for not swapping in a new power unit after the old one failed and instead relying on a spare one. Feel free to block me for some pi unit of time for my failing.

As a final note, if this is truly an act of God as more than one person posited, it is pretty convoluted and weak. A swarm of locust munching on my power cords would have been far more effective in both maintaining downtime and for the general "shock and awe" of it all.

And a few RationalWiki prods for the road:

No true Scotsman
Common descent

The expanding face of RationalWiki

2009-09-12T18:27:00.002-04:00

Awhile ago I posted a small picture of the RationalWiki server. It seemed to amuse people. Well since that time the complex of RW has expanded to take up more and more space. I decided to upload a new picture so everyone can see the new face of RationalWiki:

Woot! New network switch just arrived.

2009-09-12T12:45:00.000-04:00

I am replacing the weird little neon hunk of plastic that I think is a network switch...but the Mandarin confuses me......with a solid linksys switch I ordered from newegg. Thanks to everyone that helped donate!

I think the switch was by far the "weakest" point in the network setup, and most likely to fail next. So this is a good upgrade.

A switch should take less than a minute to install so I am not bothering with an RW intercom message. But posting this here just in case I blow something up and the site stays down longer than expected.

After this just need to get some automatic/remote server monitoring hardware and we will be set!

New status widget and update on google

2009-09-11T14:13:00.000-04:00

New widget

So for fun I have setup a little widget on the blog to show the status of the RationalWiki servers.

If the server is down it just says server down. If the server is up and working it will display the number of hours and minuets that the server has been up "straight." That means no reboot or power down. I am also displaying the 15 minute running average for the CPU load so people can see how busy the server has been recently.

Google update

So based on my searching we are back on top for Andrew Schlafly and Poe's Law which were two of our bigger hitters for search engine referred traffic. Not everything is reindexed yet but it looks like we should recover all right from our downtime.

Hardware replacement for real this time

2009-09-11T08:46:00.000-04:00

Okay, so now that our magic new backup system appears to be working, I can actually do what I meant to do yesterday. So the site is down because I am replacing hardware.

Obligatory Google prod of the day:

Denyse O'Leary in honor of the first person to openly admit to considering a defamation lawsuit against us.
Esther Hicks just because.

UPDATE: It looks like everything went exactly as planned, smooth upgrade, site back online. I will continue to monitor the situation to make sure nothing weird happens.

Hardware replacement, downtime

2009-09-10T14:32:00.000-04:00

I think I have discovered the issues with the random shutdowns that took the site off line. As is the case with everything in life the solution to the problem is going to cost money and time.

I will be heading out to purchase some new hardware this afternoon and its installation will cause some downtime. If all goes as planned it will be less than an hour. I will give a 20-30 minute warning on RW before I take things down.

Google prodding still needed btw:

Conservapeida
Gish Gallop

UPDATE: Okay I lied. I am trying to get a few things done at once, and the completion time on task 1 is taking longer than I anticipated so the hardware replacement has been delayed till late tonight or tomorrow.

Poking Google

2009-09-07T18:51:00.000-04:00

We got dropped by the search engines so I am going to poke a few of the important pages, and hope it prods re-indexing of the site:

Poe's Law
Andrew Schlafly
Conservapedia

If you have a blog or other dynamically updated website please consider poking these pages, or any of your favorite RationalWiki pages.

All systems a go

2009-09-06T19:55:00.000-04:00

Well RationalWiki should have all cylinders firing.

Sorry for the downtime, but there was naught I could do. I will be doing some thinking about how to expand remote administration of the site over the next few days. This probably means a mini-fund raiser will be in order but hopefully we can get something set up that will prevent this kind of disaster in the future.

Getting things back online now

2009-09-06T17:59:00.000-04:00

Not a hundred percent yet.

Main thing to do is some troubleshooting and then apparently importing pages that have been made on other wikis.

Database is locked for now.

Stay tuned for updates.

Update: Everything is working. NX is going to import pages from other wikis and then open up the database for the grand reunion. I am going to get food an caffeine. We have some "rebuilding" to do though since google dropped us. I will post on RW later tonight to poke people for help getting us back to awesome again.

When will we be back?

2009-08-23T15:26:00.001-04:00

I have poked at the server every way I know how. It will have to wait till I get back.

I am arriving in Toronto at 3:45 pm EST on Sunday, September 6th. It is American Airlines flight 4068 if you want to track it to know when it lands.

Add an hour to get through customs and get my baggage. I then have to catch a bus back to Hamilton. I should arrive back at McMaster around 6 pm EST. Add 30 minutes to get back to my apartment. If it is a simple solution the server should be back on by 7 pm EST September 6th. If there is something more going on I will post here by 7pm to let everyone know.

It sucks that it went down. I am disappointed but its not the end of the world. Everything will be back and working well within 14 days from now.

I hope it is not to terrible for anyone, and might provide a nice break for a few people.

If this downtime has shown RW to be of particular importance to you it might be worth it to consider tossing some financial support our way. I will use that to invest in infrastructure to allow better remote maintenance options in the future.

Pretty much worse case scenario

2009-08-19T13:17:00.000-04:00

As is obvious to most by now the site is inaccessible, however, the bot server appears to be working. This means it is not just an internet problem, but a potential issue with the server having shut down. If this is the case I doubt there is anything I can do for now.

Server down

2009-08-16T14:31:00.000-04:00

We are experience some difficulties with mysql connections at the moment which is causing the downtime.

I am trying to track down the problem as I type this.

*EDIT*

Site should be back up. Post here if you are unable to access it.

Intermitten down time

2009-07-24T14:41:00.001-04:00

I am working on backup issues, and there maybe some intermittent downtime.

Backup

2009-07-11T01:45:00.000-04:00

I am running the backup.

I am close to figuring out what was causing the problems with the automated script, and so should have a scheduled backup ready to go soon.

I was thinking around 4 am EST, let me know somewhere if you disagree.

The loop continues

2009-07-07T02:47:00.000-04:00

Backup again, then off to bed.

Here is my random/fun fact for tonight: Google drives about 5 times the traffic to our site that yahoo does. Windows live does about a third of yahoo. Other random search engines barely register.

Backup

2009-07-06T01:57:00.000-04:00

This is getting repetitive right?

That's not a bad thing since the only other reason I would have to post here is something worse has happened. I am working on redoing the automated backup scripts, they were causing some problems, and until that is done the backup is manually done before I go to bed. Which is why it is not at a set time...yet. I will get it all automated soon enough and there will be a set "known" time.

Fun fact for the day: Wednesday and Thursday are by far the most popular days for RationalWiki. Saturday is the least. Almost twice as much traffic flows through on a Wednesday compared to Saturday.

Rinse Repeat, backup alert

2009-07-05T03:05:00.000-04:00

Nightly backup is running once again.

Fun fact of the day: Less than 25 percent of RationalWiki visitors used Internet Explorer to view our site for the month of June. Almost 10 percent of our traffic comes from Linux operation systems. Not your average web surfer!

Guess what? Backup time!

2009-07-04T02:17:00.000-04:00

Nightly backup is running once again.

It will make the site inaccessible for probably 10-15 minutes, and then a little slow for another 10-15 minutes. Everything should be back to normal in about half an hour.

Fun fact of the day: The text of our little wiki is now over 10 gigabytes in size. That's a lot of "talk talk talk talk" people!

Backup time again

2009-07-03T02:08:00.001-04:00

Nightly backup is running once again.

It will make the site inaccessible for probably 10-15 minutes, and then a little slow for another 10-15 minutes. Everything should be back to normal in about half an hour.

Fun fact of the day: A google search for "esther hicks scam" or "esther hicks fraud" now returns RationalWiki's Esther Hicks article as a top rank. It is bringing in about 50-100 hits a day. Esther has a habit of attacking her attackers, maybe something interesting will come from this?

Backup time

2009-07-02T01:58:00.001-04:00

Backup script is running, said script will make the site inaccessible for 5-10 minutes, assuming nothing goes wrong.

Edit:

Backup done, everything should be back to normal till tomorrow. Night guys!

Welcome, and all is well

2009-07-01T14:58:00.000-04:00

Just adding filler, everything is working nicely.

Here is a few interesting notes that people may or may not be aware of:

*RationalWiki runs off of a cable modem connection, with about 1000 kbps upload speed. Our dedicated IP address costs me an extra $30 month which is our only real "per month" charge anymore.

*RationalWiki now has two computers dedicated to it. The first is the main server, an Intel Pentium Dual Core 2.2 GHz processor, 2 gigabytes of ram, and 300 gigabytes of space. The second is a Intel Pentium 4 2 GHz processor, 1 gigabytes ram, and 100 gigabytes of space. The second machine is dedicated to hosting bots that run on RW. At the moment it is running wpbot, stubbot, and capturebot. If anyone has a bot they would like run on our "bot" server just let me know and I can check it out.

*Thanks to recent fund raising efforts we do have power backup finally. Which is a great thing since for most of June the fuses to my room were frequently blown. Luckily, you guys were saved the majority of it thanks to our battery backup.

*Our on site backup system is protected from fire and theft by being locked in a small fire proof case drilled into my floor, connected via a USB cable drilled in a hole in the back of the case.

*External peripherals, such as monitor, keyboards, all salvaged from equipment being tossed out of my lab. This includes the second "bot" machine that was just recently setup.

And there we go, that should cover an obligatory "content" requirements for the opening of a blog. Bookmark this site and refer back to it during periods when the site is inaccessible. If I am I aware of the problem and working on it I will post here.