Restoring Drupal.org and Murphy's law
After what can only be described as the 'Extended Weekend from Hell', Drupal.org has been restored and is back in business.
First things first: what caused this outage, and why did it take so long to get the server back up? To phrase it simply and bluntly: misfortune, bad timing and miscommunication. Murphy's (annoying) law has been proven true once again.
At the moment, Drupal.org shares its resources with several other sites on a server kindly provided by Kjartan. Thursday evening, this server was hacked. One of the other sites on our server provided the hole through which the hackers entered; it appears someone wanted to turn us into a warez FTP, but completely messed it up instead. We discovered the intrusion quickly and were able to regain control of the server soon afterwards. However, the entire incident occurred only a few hours before a scheduled power outage at our current ISP; problems with remote administration and the lack of install media meant we were unable to fix the server remotely. Over the weekend we called to try and rectify the situation, but due to miscommunication with our ISP we had to wait until Monday morning before we could reinstall the OS and get the server purring again.
Still, it is not our intent to sling mud and point fingers. If anything, the whole experience has been a lesson for everyone involved.
Now, as you could read on the temporary page during the weekend, we were already planning to get another hosting, one dedicated to Drupal and related projects. Talks have been going on with Scott Kveton of the Open Source Lab for several weeks: they will provide everything we need for hosting and support, as long as we provide the machines. For this we planned a fundraising event which took place this weekend. The fundraiser was a great success, and during the next few weeks we will be gradually migrating to the new servers and new hosting facility at OSL!
As the amount you managed to raise for Drupal was above and beyond our expectations, we have asked the OSL to come up with a complete infrastructure plan for Drupal.org and its related services which will optimize reliability. Rest assured that your money will be well spent. Stay tuned for more information about this.