Wednesday, September 23, 2009
When it rains...
So God's will has struck again, the site is down but this time it has nothing to do with the server. My internet connection seems to have crapped out. Because I pay $200 a month for it though one of the perks is "emergency service" so I have a guy coming in sometime beteween 5pm-8pm EST to take a look at it and hopefully get everything back and running.
Tuesday, September 22, 2009
No sleep for the server admin
So I was up till 4 am trying to get the fault protection working. It appeared to be working beautifully so I went to bed. Only to be awoken 3 hours later with it going nuts. I have removed it from the system for now. I think I need another piece of hardware to get everything working together the way I want. So I am putting it all on hold for now. I will order the new hardware today so it could be upwards of a week before I go at this again.
My goal is to have it all in place before my trip to Chicago.
And now I think I am getting a cold. I blame the stress and lack of sleep damn it.
My goal is to have it all in place before my trip to Chicago.
And now I think I am getting a cold. I blame the stress and lack of sleep damn it.
Okay I lied
As is probably obvious I lied in the last updated when an idea for a solution to my problem came to my mind.
So I think I have everything setup. I am going to avoid going into specifics for security reasons but we now have much greater remote administration abilities that are no longer dependent on the server being online. I have also setup a range of automated monitoring software and utilities that will aid in both keeping track of site up time and doing some automated tasks that should allow the server to recover from all but the most serious of crashes automatically.
With that I am going to bed.
So I think I have everything setup. I am going to avoid going into specifics for security reasons but we now have much greater remote administration abilities that are no longer dependent on the server being online. I have also setup a range of automated monitoring software and utilities that will aid in both keeping track of site up time and doing some automated tasks that should allow the server to recover from all but the most serious of crashes automatically.
With that I am going to bed.
Monday, September 21, 2009
Extended maintenance
The site is going down this afternoon for extended maintenance. Running some tests, changing some options and install some new hardware. All designed to try and help deal with some of the recent outages. Running the backup first, then I will get started.
Update: Screw it I am done for tonight. Got about half of what I wanted figured out. Luckily the last half lets me keep RW up most of the time I am working. I will have to come back and keep working on this probably tomorrow which means don't freak if there is intermittent downtime for a minute or so every now and then for the next day or so.
Update: Screw it I am done for tonight. Got about half of what I wanted figured out. Luckily the last half lets me keep RW up most of the time I am working. I will have to come back and keep working on this probably tomorrow which means don't freak if there is intermittent downtime for a minute or so every now and then for the next day or so.
Repairing a table in the database
Things will be locked up for a few minutes while the repair is run. I am aware of the situation and working on it, and hope to have things back up shortly.
UPDATE: Repair is done site is back online let me know if there are further issues.
UPDATE: Repair is done site is back online let me know if there are further issues.
Sunday, September 13, 2009
Server crash post-mortem
Time for the official post-mortem of what happened as far as the server crash goes. The official cause of the crash shall be listed as a failing power supply unit.
About a month ago the power supply for the server went completely dead. In order to get the server back up and running as quickly as possible I swapped in a spare unit I had from an older computer. It did the job beautifully. Seeing as how everything appeared to be working fine and there were no substantial problems I didn't replace it with a new unit.
About 4 days before I left for my trip back home the server shut itself down. When I booted it back up I had some problems with the MYSQL server and so chalked the problem up to that. Then I left and went away. We all know what happened next. When I got back to the server I found that it was in the same disabled state as the first crash. I got it back up and decided to watch and see what it would do. A day later the same thing happened.
So I ran some tests on the power supply and it was providing irregular power on the 12v rail, my guess is that was probably leading to a temperature triggered shutdown. Anyway, regardless, 2 days ago I purchased a high-end power supply unit and swapped it in. Everything seems to be running fine now.
Thanks to the donations everyone at RW gave or have promised to give, I have gone ahead and upgraded some of the networking hardware that was worrying me as well. I am also working on getting some hardware to allow for remote management of the server even if it is unresponsive, as well as server resets automatically if it becomes unresponsive.
So the whole thing is my fault for not swapping in a new power unit after the old one failed and instead relying on a spare one. Feel free to block me for some pi unit of time for my failing.
As a final note, if this is truly an act of God as more than one person posited, it is pretty convoluted and weak. A swarm of locust munching on my power cords would have been far more effective in both maintaining downtime and for the general "shock and awe" of it all.
And a few RationalWiki prods for the road:
No true Scotsman
Common descent
About a month ago the power supply for the server went completely dead. In order to get the server back up and running as quickly as possible I swapped in a spare unit I had from an older computer. It did the job beautifully. Seeing as how everything appeared to be working fine and there were no substantial problems I didn't replace it with a new unit.
About 4 days before I left for my trip back home the server shut itself down. When I booted it back up I had some problems with the MYSQL server and so chalked the problem up to that. Then I left and went away. We all know what happened next. When I got back to the server I found that it was in the same disabled state as the first crash. I got it back up and decided to watch and see what it would do. A day later the same thing happened.
So I ran some tests on the power supply and it was providing irregular power on the 12v rail, my guess is that was probably leading to a temperature triggered shutdown. Anyway, regardless, 2 days ago I purchased a high-end power supply unit and swapped it in. Everything seems to be running fine now.
Thanks to the donations everyone at RW gave or have promised to give, I have gone ahead and upgraded some of the networking hardware that was worrying me as well. I am also working on getting some hardware to allow for remote management of the server even if it is unresponsive, as well as server resets automatically if it becomes unresponsive.
So the whole thing is my fault for not swapping in a new power unit after the old one failed and instead relying on a spare one. Feel free to block me for some pi unit of time for my failing.
As a final note, if this is truly an act of God as more than one person posited, it is pretty convoluted and weak. A swarm of locust munching on my power cords would have been far more effective in both maintaining downtime and for the general "shock and awe" of it all.
And a few RationalWiki prods for the road:
No true Scotsman
Common descent
Saturday, September 12, 2009
The expanding face of RationalWiki
Awhile ago I posted a small picture of the RationalWiki server. It seemed to amuse people. Well since that time the complex of RW has expanded to take up more and more space. I decided to upload a new picture so everyone can see the new face of RationalWiki:
Subscribe to:
Posts (Atom)
