This post is part of a series to inform the Drupal community about the drupal.org redesign project, and the work the Drupal Association is funding to help get the redesign completed. If you would like to contribute to the redesign as a volunteer, see the community initiatives redesign page. If you'd like to contribute to the redesign financially, see the Drupal Association memberships and donations pages.

Project management changes

Since our last update, we’ve been busy with the following changes :

• We are now running the Drupal.org redesign project in two week sprints. Each sprint begins with a planning meeting and finishes with a demo meeting. During the planning meeting, the contractors and project managers set out tasks to complete during that sprint with an effort score for each task.
• We have Skype check-ins with the contractors three times per week.
• The project managers meet separately for 1 hour per week to discuss resources, coordination, and budget issues.
• Key redesign team participants can assign issues in the drupal.org issue queues (and have issues assigned to them). This functionality was chosen over moving an external (and therefore less open and transparent) project management system
• The redesign issues are in many different issue queues, so we initiated the drupal.org redesign tag last year. We are also interested in tracking progress by sprint, as well as the overall visibility by type of work. Therefore, issues are now tagged more thoroughly, for instance, a single issue will be tagged drupal.org redesign, drupal.org redesign sprint 1, drupal.org redesign solr or drupal.org redesign, drupal.org redesign sprint 2, drupal.org redesign content
• Revamped and improved the community initiatives redesign page to provide clearer starting points to those who would like to help.

Project management accomplishments

• Recruited theme implementers, and Git contributors
• Created a detailed 1000 hour Git migration plan
• Organized standard meeting infrastructure, shared calendars, conference lines, etc
• Worked on new hosting ads redesign
• Finalized the content migration plan

In an ideal world...

Ideally, a small team of experts would have been able to implement the full redesign in their spare time, but during the last two years we realized this was not pragmatic. Some companies offered to implement entire parts of the redesign, but the dependencies on infrastructure, search, and project module proved too difficult without a dedicated architect, and dedicated support in those areas. Our expert volunteers from the infrastructure team are in high demand and their contributions often focus on maintenance and innovative new capacities that are developed without a schedule. For the redesign, we needed to involve a wider implementer group, and that has meant we’ve had to build out a new drupal.org infrastructure and development process.

The drupal.org redesign is different from other projects in the following ways:

• Most Drupal development projects are done by relatively small teams of full-time, paid developers. Our project teams big, very geographically dispersed, and up until recently, 100% volunteer.
• These contributors work for free (the exception being our contractors), but are personally motivated to improve drupal.org, our community's home.
• We’re empowering 50+ people to contribute to the development of drupal.org
• The infrastructure team is a small team of highly competent volunteers who are granted access based on established trust and expertise built up over years
• We have a responsibility to deal with access control, privacy, and security when offering volunteers access to private databases, themes, and development, and staging infrastructure

Infrastructure update

In the past, the code relating to the drupal.org infrastructure has been held in a variety of version control repositories and external systems, depending on the team responsible for it. Thanks to the efforts of David Strauss who was willing to step up to handle implementation and maintenance, we now have a single, cohesive BZR code repository for all of our infrastructure efforts.

To allow developers to freely work on development and staging infrastructure we’ve allowed their SSH accounts access to Hudson. Hudson monitors executions of repeated jobs. Many of the jobs that developers need to do are repeatable and a common subset of development tasks such as checking out code, updating a database, deploying a new site, or creating a new Solr search index. Hudson allows us to give a wide variety of developers the ability to do common development tasks without tripping over each other on a shared infrastructure. See http://wiki.hudson-ci.org/display/HUDSON/Meet+Hudson for more information.

How Drupal.org is built

Of course, most developers want to be able to work locally. Automating the creation of drupal.org database copies that have been scrubbed for privacy and security has been a top priority for a long time. A lot of the redesign is about implementing search-related features. It is also important to provide scrubbed Solr indexes to download. If a developer were to accidentally delete their local copy of the entire drupal.org Solr index, it could take days to re-index drupal.org on a local computer before they could get the live results they need to keep working on a search feature.

Infrastructure accomplishments in Sprint 1

For this sprint, Narayan and Rudy from Tag1 consulting focused on implementing Hudson jobs, providing BZR access, making databases and providing a Solr index.

Solr: Making it easier to find what users want

In order to make it easier to find projects on Drupal.org, the redesign emphasizes project-based search forms in project sections and search results. For example, people should be able to sort the main modules page for Drupal 7-only modules. The project and Solr developers decided to abstract out their search forms to be re-used in the Download and Extend sections as well as for project-based search results.

Example search form

The common search form for search results:

And project browsing:

They also decided that displaying faceted search results in blocks was generally useful enough to make this part of the base Apache Solr project. For project module-specific searches, like “Most installed” and “Most downloaded,” we added these facets to the Solr project module. These results would then just show up in a faceted search result block for the Download and Extend pages, or the search results page.

If the search results are faceted on the project content type, then we automatically make a project search form available for further searching.

Once project searching is implemented we are going to be focusing on multi-site search. The proposed plan is to have all search results appear on drupal.org, no matter which sub-site you search on. If the user wants to search only the current sub-site, a tabbed search form will be available.

Solr issues for Sprint 1

• #865510: Add project_solr_browse_projects_form as a block on the main search page when a project type facet is selected
• #867490: Call modify_query for drupalorg_order_facets
• #862240: Provide additional solr facets for advanced project browsing
• #867312: Adding Solr content for the main download page
• #867526: Multi-site search functionality
• Documented what themers can/can't do

Project module

The Project module is the most important software in the Drupal project as it allows the community to collaborate to build Drupal core and the 8700+ contributed modules, themes, translations and installation profiles which make Drupal so successful.

A key improvement of the redesign is to make discovery of projects easier. In order to improve the Project module, several facets were added the Solr Project module which allowed for “Most installed, Most downloaded” search results. These are now part of the new project-type landing pages.

In preparation for the Git integration the project module will now lock the project shortname as the repository directory name.

Project module issues worked on sprint 1:

• #781300: option to lock project short name
• #868794: Add an apachesolr_result_block module to render solr facet results as blocks
• #867312: Adding Solr content for the main download page

Architecture

Another huge challenge with the redesign has been explaining the development standards and development culture that drives the development of a large community site. Drupal developers who have participated in module, theme, or core development have learned “The Drupal way”. We quickly realized that a huge portion of the redesign effort was explaining “The Drupal.org way” to volunteers.

To help explain the drupal.org site development culture we documented it.

• Drupal.org development guidelines
• How Drupal.org is built
• How Drupal.org uses Drupal

Community initiatives - how you can get involved

In order to bring focus and consistency to the technical implementation, the redesign architect has listed all the parts of the redesign that are not currently being implemented by the redesign contractors. In addition to themer and developers, we could use people with content skills, particularly IA.http://drupal.org/community-initiatives/redesign

Paying for the redesign

The Drupal Association currently has five contracts that involve as many 8 paid contractors working on the redesign, not including the association’s general manager. We also rely on ongoing volunteer contributions from the project management team, the infrastructure team, and the redesign implementation volunteers.

In the last 6 weeks the Association has spent approximately $60,000 on the drupal.org redesign, even with significant volunteer contributions. In order to complete the redesign, the Association will be spending well over $200,000. This money is coming from donations, memberships, affiliate programs, DrupalCon sponsorships, and advertising.

The Association plans to increase the quality of advertising available in the sections of drupal.org which currently have advertising and expand advertising to additional sections, as defined in the advertising section of the redesign prototypes. The advertisements will remain contextual, relevant, and useful to the Drupal community. The goal is to make improvements to drupal.org sustainable and empower the community to make drupal.org a home worthy of the Drupal community.

What's next?

• Sprint 2 is now under way and on schedule. Major work items include removing blockers to the Git migration from Project module, further enhancements to the underlying functionality for the Download & Extend pages, further infrastructure automation, and establishing AMI-based infrastructure to enable Git migration volunteers to help more efficiently.
• At Drupalcon Copenhagen, we will be talking about The Future of Drupal.org and doing a Redesign BOF
• We look forward to having more community members step forward to take advantage of our new processes for contributing

Where can I see what's done in action?

Head to http://redesign.drupal.org/ (username: drupal, password: drupal)