Traffic spike post-mortem

As many of you are already aware there was a huge traffic spike to RW over the weekend. One of our tumble weed articles (Scientific evidence of evolution being a hoax) hit the big time, again actually.

At its peak on Sunday we had made the front page of Reddit, all over Facebook and several large blogs. Over 100,000 people accessed our site during this time and the site became almost impossible to use.

I understand the frustration that this can potentially cause, and several people initiated discussions about what to do to allow RW to withstand these kinds of spikes.

My response to this is that it doesn't make sense to develop our infrastructure using this weekends traffic loads as our upper bound tolerance level. This is the single largest traffic spike in RW history, and traffic spikes big enough to seriously impact the performance of the site happen only a few times a year. The infrastructure we would need to be able to keep RW working without impact under that kind of traffic load would likely increase our costs 10 fold.

Our existing infrastructure supports our average traffic load, and can support surges several times what we get as baseline. When are greatest limiting resource is financial we have to be smart about how we spend our money. It is better to struggle a few times a year under extreme traffic load, then to go broke in a month and have to shut down operations.

With that said, I am constantly monitoring server performance, and the increase in baseline traffic that we get. Our baseline traffic is continuing to grow at a significant rate, and the rate its growing has been constant for over a year. As the baseline demands increase, I have moved and altered the server infrastructure to support it within the financial limits we have.

The hope is that as our requirements increase, so does the pool of users that are willing to help us out with a few bucks along the way. Thus allowing us to expand our infrastructure as needed. So far this is proving to work out well. We are in the middle of a current fund raiser on RationalWiki that is on track to be our most successful yet. If you haven't had a chance to toss a few dollars our way, now would be a great time!

Ultimately, I think we should celebrate our growing popularity, and remember that growing pains are signs of just that: growth!