Saturday, October 3, 2009
An act of "nature"
Sorry folks, based on the blinking clock above my computer it appears the power went out to my house. All the various fault protection equipment kicked in as appropriate but I can't sustain the server for longer than about half an hour with no power. Based on site monitoring it appears that it was inaccessible for only a few minutes at most.
Friday, September 25, 2009
Fault protection take 2
I have received the additional hardware I needed to get the fault protection working...I think. I will be setting that up and running some test on it today. It shouldn't effect RationalWiki at all as I can do my testing further down the network chain. If all goes well I will need to restart the server and that's about it. I will drop an intercom on RW when/if I do that.
Update: Okay, fault protection seems to be working well. I have implemented it live on the server now. I will continue to monitor everything and adjust as needed. But all appears well for now.
Update: Okay, fault protection seems to be working well. I have implemented it live on the server now. I will continue to monitor everything and adjust as needed. But all appears well for now.
Wednesday, September 23, 2009
So just how bad has it been?
The recent downtime issues annoy me at least (probably more) than every user of RationalWiki. As always, I strive to offer the best service as far as performance, availability, and protection against catastrophe that I can. But I am not perfect, and I think it is important to keep in mind that this is hobby, and my "real life" as a graduate student is as overworked and under paid as most cliches depict it.
All that said and done, just how bad has it really been? After the great crash in August I started working with a service to help monitor RationalWiki's up time and server performance. With close to three weeks worth of data we can paint a picture for how bad things have been.
If you take a look at this analysis you can see that RW has actually been up 95 percent of the time. That is not bad all things considered. Take into account a few points: first that a good chunk of our downtime is packed together so it is mostly caused by 1 or 2 disasters that caused prolonged downtime, and second that our nightly backups cause time out errors for about 20-30 minutes. If you remove the few major disasters our uptime averages just over 99 percent a day, and without the backups you are looking at 100 percent coverage most days.
The key then is disaster recovery. To be able to quickly handle issues that cause long protracted downtime. Most of these are easily handled if I am awake and with in walking distance of the server. The issues today with the cable going down are very rare. So we are left with one major issues: server cop-outs that prevent remote log in and shutdown the site that occur when I am either asleep or traveling.
I am actively working on a solution that I think will greatly increase the servers ability to auto-recover from failure, and to expand the options for remote administration in the event of catastrophic failure when I am not present (ala what happened in August).
A lot of this is trial-by-error and learning as I go. I have never done a project like this before. All we can do is learn from our mistakes, and move forward with the goal of doing the best we can. That said what we do have is pretty good I think.
All that said and done, just how bad has it really been? After the great crash in August I started working with a service to help monitor RationalWiki's up time and server performance. With close to three weeks worth of data we can paint a picture for how bad things have been.
If you take a look at this analysis you can see that RW has actually been up 95 percent of the time. That is not bad all things considered. Take into account a few points: first that a good chunk of our downtime is packed together so it is mostly caused by 1 or 2 disasters that caused prolonged downtime, and second that our nightly backups cause time out errors for about 20-30 minutes. If you remove the few major disasters our uptime averages just over 99 percent a day, and without the backups you are looking at 100 percent coverage most days.
The key then is disaster recovery. To be able to quickly handle issues that cause long protracted downtime. Most of these are easily handled if I am awake and with in walking distance of the server. The issues today with the cable going down are very rare. So we are left with one major issues: server cop-outs that prevent remote log in and shutdown the site that occur when I am either asleep or traveling.
I am actively working on a solution that I think will greatly increase the servers ability to auto-recover from failure, and to expand the options for remote administration in the event of catastrophic failure when I am not present (ala what happened in August).
A lot of this is trial-by-error and learning as I go. I have never done a project like this before. All we can do is learn from our mistakes, and move forward with the goal of doing the best we can. That said what we do have is pretty good I think.
And we are back....
What appears to have happened is my next door neighbors got a cable hookup, and God put it into the mind of the cable technician to accidentally disconnect my cable while doing the hook up.
Where God failed in his plan was in prevent another technician from showing up and fixing the problem. An even larger failure was that he allowed the technician to show up hours early! How often does that happen? I think we know who to thank for that!
On a more serious note, in my comment on the last post I discussed the monthly cost of running RW. The point was not to ask for more money, the cost stems form the dedicated IP address and extra bandwidth that RW requires. It is an extra $50 a month above what I would pay anyway. I get about $20 a month in donations from people that give every month. That means my per month out of pocket expense is $30 a month. I can totally handle that. The occasional need for new hardware that I can't afford is usually met almost immediately by a small donation drive. Every now and then I get an extra donation that I keep on the back burner for emergencies.
The RW accounts have about $80 sitting in them right now for emergency purchases. I like to keep that amount around $100-$150 to make sure I can get almost anything we need.
All of this is to say that at the moment the financial cost of RW is not really a burden on anyone. That was the point to moving to a privately hosted site.
If we were to try and move to a commercially hosted site the financial burden increases multiple fold for me and those people who are able and willing to donate. There is also substantial risk that if we fail to get enough cash we could lose the site.
So we put up with this less than perfect up time because it means that RW is under no immediate risk of permanent shutdown and is not a going to bankrupt anyone in the proccess.
Where God failed in his plan was in prevent another technician from showing up and fixing the problem. An even larger failure was that he allowed the technician to show up hours early! How often does that happen? I think we know who to thank for that!
On a more serious note, in my comment on the last post I discussed the monthly cost of running RW. The point was not to ask for more money, the cost stems form the dedicated IP address and extra bandwidth that RW requires. It is an extra $50 a month above what I would pay anyway. I get about $20 a month in donations from people that give every month. That means my per month out of pocket expense is $30 a month. I can totally handle that. The occasional need for new hardware that I can't afford is usually met almost immediately by a small donation drive. Every now and then I get an extra donation that I keep on the back burner for emergencies.
The RW accounts have about $80 sitting in them right now for emergency purchases. I like to keep that amount around $100-$150 to make sure I can get almost anything we need.
All of this is to say that at the moment the financial cost of RW is not really a burden on anyone. That was the point to moving to a privately hosted site.
If we were to try and move to a commercially hosted site the financial burden increases multiple fold for me and those people who are able and willing to donate. There is also substantial risk that if we fail to get enough cash we could lose the site.
So we put up with this less than perfect up time because it means that RW is under no immediate risk of permanent shutdown and is not a going to bankrupt anyone in the proccess.
When it rains...
So God's will has struck again, the site is down but this time it has nothing to do with the server. My internet connection seems to have crapped out. Because I pay $200 a month for it though one of the perks is "emergency service" so I have a guy coming in sometime beteween 5pm-8pm EST to take a look at it and hopefully get everything back and running.
Tuesday, September 22, 2009
No sleep for the server admin
So I was up till 4 am trying to get the fault protection working. It appeared to be working beautifully so I went to bed. Only to be awoken 3 hours later with it going nuts. I have removed it from the system for now. I think I need another piece of hardware to get everything working together the way I want. So I am putting it all on hold for now. I will order the new hardware today so it could be upwards of a week before I go at this again.
My goal is to have it all in place before my trip to Chicago.
And now I think I am getting a cold. I blame the stress and lack of sleep damn it.
My goal is to have it all in place before my trip to Chicago.
And now I think I am getting a cold. I blame the stress and lack of sleep damn it.
Okay I lied
As is probably obvious I lied in the last updated when an idea for a solution to my problem came to my mind.
So I think I have everything setup. I am going to avoid going into specifics for security reasons but we now have much greater remote administration abilities that are no longer dependent on the server being online. I have also setup a range of automated monitoring software and utilities that will aid in both keeping track of site up time and doing some automated tasks that should allow the server to recover from all but the most serious of crashes automatically.
With that I am going to bed.
So I think I have everything setup. I am going to avoid going into specifics for security reasons but we now have much greater remote administration abilities that are no longer dependent on the server being online. I have also setup a range of automated monitoring software and utilities that will aid in both keeping track of site up time and doing some automated tasks that should allow the server to recover from all but the most serious of crashes automatically.
With that I am going to bed.
Subscribe to:
Posts (Atom)
