At DrupalCon Chicago this past week, there was a "Core Conversations" session track, made up of sessions pitched by contributors to the core Drupal project. A wide range of topics were covered from the Butler project (a new system for context and blocks), to the built-in Help system, to Deployment strategies, to redesigning the issue queue. These sessions were shorter presentations followed by a discussion period for the attendees to give input on the topics.

The final conversation on Thursday by Dries Buytaert (Drupal's project lead) focused on discussing the development process for Drupal 8. (It was also exciting to witness the Drupal 8 branch being created live during the session!) During the presentation portion, Dries described in more detail the process changes he had suggested during his keynote. He then opened the floor up for everyone to bring up issues they felt needed some attention/discussion.

Discussion points

Here is the list of discussion points (that Dries noted during the talk) we core enthusiasts came up with:

• More structured sprints - project management
• Sandboxes (aka. Samboxes, since Sam Boyer was a huge contributor to Drupal's CVS to git migration) and locations
• Timeline of release cycle
• Hybrid development cycle
• Ubuntu model?
• Gate details; performance testing, etc.
• Roles of initiative owners
• Backporting
• Non-initiatives / small changes / bugfixes -- different process?
• Tools for usability/accessibility
• Process around new initiatives / proposal
• Documentation gate - workflows

Many, but not all points were discussed, and as we progressed through the conversation, I began to see parallels between some of the process changes we've implemented here at Affinity Bridge and what's going on in the Drupal development process. When I first began my position as Project Manager here, the first task I was assigned was to figure out how to make our development process work better, especially for larger, ongoing projects. This was sparked by the pain of working on very large projects, with huge issue backlogs and many feature requests, and no set launch date. 

After a lot of reading and research, our team started experimenting with a more agile process. Picking smaller chunks of work, completely finishing them, picking a new chunk of work, completely finishing it, etc. helps to make progress faster and at a better quality. Many times now, the benefits of this process have proven themselves, and I am beginning to see how some of the main points we discussed at the core conversation could be addressed with some incremental changes towards a more agile development process. 

A sidenote, as I was discussing this over the weekend, Bruno De Bondt from DeWereldMorgen.be pointed me at a blog post from one of his old coworkers from Krimson (a development shop in Belgium), Kristof De Jaeger (aka. swentel). The post is from about a month and a half ago, and outlines a proposal for an agile Drupal 8 development process. He goes into much more detail about the process and different roles involved than I will here, but it is well aligned with my thoughts (and potential long term vision), so I recommend you read it!

Specific ideas about sprints and agile for core

Before I delve into details here, I would like to preface this by saying I am not a religiously devoted agile follower. So far, what I've seen work is using elements of the agile methodology that fit for the team and project. Some devoted capital-A "Agile" followers might find that to be a flaw, but I don't think it's necessary to shoehorn people into a process. It's important that anyone adopting this feel comfortable with it, and not forced into it. Also, speaking briefly with Dries after the core conversation, he noted that it's important with such a large community not to try and undertake too much change at once (referring to the git migration and also the new idea of core "initiatives"). I very much believe those are wise words, and so am more keen to suggest some practices that the community can experiment with. If these are successful, then perhaps they can be applied more overarchingly.

Now, to go through a few of the main discussion points and others that came up, and relate them to how a more sprint-based/agile process might help address them.

New opportunites from sandboxes and git

The move to git is obviously going to open up many doors as far as more collaborative work and changes to workflows. Any Drupal.org user can now have their own development sandbox, and as Dries mentioned anyone can now fork core (ie. make a duplicate version and add improvements to it), though it's not always recommended! 

As many others noted in the session, this is going to allow multiple people to easily work on different parts of core and then merge their work back together. It could also allow multiple people working on different approaches to the same parts of core, to then compare and combine the best pieces. In any case, it will allow for a lot more safe experimentation and collaboration.

Keeping the criticals count below 15

This was something Dries brought up in both his keynote and the session. The Drupal 7 release cycle was very drawn-out because of the large quantity of criticals accrued through the three years of work. Longer release cycles tend to lead to burn-out and work that doesn't get fully completed.

Having the sandboxes and git will allow for a totally new approach here, one which is common to a more "agile" process, which prioritizes keeping the master branch clean and ready to launch at any time. To work like this, the master branch for core would be kept in a launchable state, and any changes or new work would be done in branches/sandboxes which would have to be completely done to be merged back into core. The "gates" would protect the master branch from bugs or incomplete work, so that we could be relatively certain that core is always ready to ship.

As a result, we wouldn't be aiming for under 15 criticals, but rather a constant of zero criticals in the master branch.

Small changes and bug fixes

Small changes and bug fixes would also be done in branches, but merged back in more frequently. There would be no necessity to save up batches of small fixes, though they could be done in batches related to different pieces of functionality. 

We would want to aim to avoid incurring technical debt, ie. fix problems as we go, as much as possible. Never delay working on bugs that will need to be fixed before launch. If that's the case, the work is not "done" and should not go into the master branch. This may mean more refactoring, but it helps keep the master branch ready to ship at all times.

Gates

The "gates" Dries talked about are still being defined, but the general idea is that there would be some steps that need to be completed before any branch would be merged into the master. Examples of what gates there might be include:

• Documentation: making sure all of the code/API documentation is complete/updated, and ideally make sure the online documentation is also complete/updated. Possibly could go through review by Documentation Team members or other developers.
• Accessibility: making sure that basic accessibility standards are met. Possibly could go through review by Accessibility Team.
• Testing: making sure automated tests are written and pass. Possibly could go through manual testing as well.
• Performance: making sure performance meets certain standards.
• Design/UI: having design or usability reviews.

Making this concept of "gates" work will require defining a set of requirements and standards, and possibly finding people willing to do frequent reviews. But it will also mean that this additional work is really done before merging into the master branch, and that "done" refers not only to code being complete, but a more holistic interpretation of when work is ready to ship.

Timeline of release cycle

Dries suggested a shorter release cycle for Drupal 8, with likely a more focused and smaller set of initiatives (which he's outlined as areas of focus that will have initiative leaders). Keeping the master branch ready to ship at all times will mean that the release cycle can be as short or long as we want, and that we are not limited by half-finished functionality in the master branch (since only finished functionality would ever be merged in).

More structured sprints and phases

A lot of these goals and ideas lend extremely well to working within more structured sprints, and using phased development for larger initiatives. My initial suggestion (Kristof's post suggests two month sprints) would be for one month sprints. Despite the issue of working with a spread out team of mostly volunteers, I find that shorter sprints lead to better momentum. I also feel that it will be easier to pick out clear sets of issues to work on per sprint if they are relatively short.

Another benefit to shorter sprints would be the potential to attract more help from people who aren't interested/able to work on core more consistently. It's a lot easier for someone to commit to working on some functionality for a month rather than for three years. I would bet that once someone helped with one sprint, they would be far more likely to end up helping on another one down the road.

For larger pieces of work, we'd want to work in phases that are several sprints long. Each phase would have a set of functionality that can be merged with the master branch when complete. And the end of each phase would be a great place to work through the "gates". It would be nice to think we could go through the gates at the end of each sprint, but I don't think we currently have the resources to do this. It might take many phases to get an initiative fully complete, but the length of the initiative wouldn't necessarily delay a potential release.

A note on infrastructure...

Structured sprints would be much easier to do if we were able to add "sprint" and/or "phase" fields to the issues, but even without this we can always organize what is in each using tags, as was done with the git migration.

It would also be ideal to be able to relate issues in a richer way like in some issue tracking systems such as Unfuddle (see image below). Despite Unfuddle not being created specifically for agile development, it's easy to use it this way. I tend to create "parent" tickets for meta-issue/discussions and then create specific work item tickets as "children". You can also mark "related" issues, which aren't part of the batch of tickets, but are somehow related. And you can have parent tickets that have child tickets that are parents to others, so it's possible to create a hierarchy of what needs to be done to move onto another piece of work. Then, I create a "Milestone" for each sprint, with start and end dates, and add and prioritize tickets in the milestone appropriately after discussing priorities with the product owner and the development team.

If this interests you, there are issues filed for redesigning the issue queue and creating functionality to support meta-issues that you can add your opinions to.

Project management

This all seems to beg the question: does the Drupal project need a dedicated project manager (or scrum master)? 

At this point, I would say no. Between Dries (who tends to fill the product owner role), the core maintainer he appoints (who tends to act somewhat as a project manager, and does a lot of QA), and now the new Initiative leads, I don't feel that this would be necessary during a period of more experimental adoption.

If teams and individuals try this method, and find it works well enough that it could be adopted fully for all core development, it would be a very good idea. A much more structured process takes a lot of work to keep organized and stick with, and I believe it's best to keep developers' time free to be dedicated to development as much as possible. So at this point we would want to create some overall structure in the issue queues to be able to manage sprints efficiently, and have someone who can oversee the project's organization as a whole.

Realistically, I don't think this would be necessary very soon. For now, I would suggest anyone interested start with:

• Running set sprints (recommending one month). Meeting at least once per sprint (on IRC or by voice) to review and align priorities amongst whoever is working on a given issue/set of issues/initiative. Communicating frequently on IRC (like we always do!).
• Defining work to be done at the start of the sprint, and doing your best not to introduce new work into the in-progress sprint. In the beginning, underestimate how much can be done, until you get a feel for the momentum possible in a single sprint. If some things don't get finished, they go back into the "backlog" (or main queue of issues) and are reevaluated in regards to whether they will be in the subsequent sprint.
• Making sure at the end of the sprint (or phase for larger initiatives), that you pass through the "gates" and deliver functionality that is completely "done" and ready to merge into the master branch.
• Making sure that no bugs get into the master branch so it can always be ready to ship. As a result being able to make the release cycle any length, and launch on short notice if desired.
• Avoiding technical debt (things that you put off that will need to be done later); making time for refactoring or bug-fixing as you go. Possibly even having a technical debt recovery sprint early on.
• Defining the "gates" clearly and deciding who is responsible for signoff before merging into the master branch of core.
• During this experimental phase, making sure there are some resources that teams and individuals can refer to and learn more about these methods. Possibly looking for project managers in the Drupal community who are familiar with agile development, and open to informal advising on IRC.
• Remembering that this is a pilot project of sorts. Not putting too much pressure to follow capital-A Agile strictly. Letting this be an organic process to see what fits the community best, while leading to a more efficient, clean, and smooth process.

I was extremely happy to hear how open and interested the other attendees of Dries' core conversation were, and hope this helps clarify some ideas. This should be an open discussion rather than anything too prescriptive, so I would love to continue the conversation and hear your feedback, questions, and concerns below.

(ps. The Drupal 8 wordmark used with this post on the homepage is from Dries' slides, just for the record!)