Measuring Drupal's health

The Drupal community is diverse. We're from different places. We speak different languages, code in different languages, and use Drupal in different ways. But the one thing we all have in common is that we care about the fate of Drupal. We invest our time, intellect, and emotions into this project. And we want that investment to make the project successful.

We have a lot to be proud of. In the last few years, Drupal use and community contribution has grown tremendously. Some metrics—like comments and commits created each month—are available at drupal.org/metrics. We share project usage and track BuiltWith statistics. We've also started reflecting company contributions in the marketplace and on the organizations list.

The Association's mission is to unite a global open source community to build and promote Drupal. One of its 3-to-5 year goals is to ensure the sustainability of our project and community. To meet that goal, we need to get much better as a community at answering the question: Is Drupal healthy? But that's not an easy question to answer.

In the last board meeting, board members and staff talked about what kinds of metrics show health. In this post, I share some of that thinking, and ask for your ideas. But before we get into recommendations, let’s start at the beginning.

What's project health?

Association staff and board members use “project health” as shorthand for project success. "Health" is robust. Think about your own health. You might think first about physical aspects (Do I have a fever?). But you also might think about emotional aspects, like depression or stress. You may even think about social aspects (Do I have the right people around me to support me and keep me focused on the right habits?). Health isn't binary. Health is multi-dimensional.

So, too, is the health of an open source project. It's easy to take the temperature of a project with metrics like usage or number of committers. But to understand the complexity of our project's health, we need broader measurements. In our work on staff, we've defined four dimensions of product health1 we think we need to explore:

  • Product: whether the business of the Drupal product (the software) is sound. Examples of areas to explore: marketshare; Drupal businesses' revenue.
  • People: whether we have the right kinds of people contributing the right kinds of things. Examples of areas to explore: number of contributors; kinds of contributions; contributors' skills.
  • Process: how the way we do things contributes to the project. Examples of areas to explore: ability to meet published release dates; succession planning.
  • Systems: whether we use the right tools. Examples of areas to explore: testing times for DrupalCI; amount of documentation edits; responses to posts in forums; issue resolutions; commits and integrated repositories.

Acknowledging our limitations

There's the ideal, and then there's reality. We know we won't be able to track everything we want to. So, we'll have to make choices. We'll have to choose metrics that give us directional, even if not precise, accuracy. So, what are some of our limitations?

The future of the project

Knowing what categories of metrics we need to track is necessary, but not enough. Setting metrics also requires knowing where we want the project to go. Taking your temperature is only helpful if you know what you want it to be (and what it means if it's not).

This is a challenge Association staff discussed with the board. The Association doesn't set the software's future. It does, though, need to know where the project is going. To support the project's journey, it needs to know its direction. For example, maybe headless Drupal is the future. If so, we might measure “People” success by how many JavaScript developers contribute. But it’s not 100% clear what the future holds for Drupal. We have work to do before we can identify the best metrics.

Resources

Tracking project health is an Association priority, but it’s not its only mandate. It has to consider the time and expense invested in DrupalCons and Drupal.org too, for example. Unfortunately, budget limitations mean not hiring analysts or consultants to help. For the most part, they mean working with the (wonderful) people already on staff.

So, for example, measuring contribution only by code or comment attribution isn't enough. People contribute in so many ways. (There's even an issue open on this topic already.) Will measuring contribution expand to include other things? Yes. But will it also likely still not give some contributions the attention they deserve? Unfortunately, yes. Hard choices will mean we'll all have to accept some less than ideal outcomes.

Balancing competing frames and other fun factors

Once we know what outcome we want and have found things we can actually measure, we'll still need to do more. Any metric we choose has to also align with the project's mission and our community’s values. We also can't be too dependent on internal metrics. We'll have to measure our success with external indicators too. (See? There's a lot to think about here!)

Examples, please

There’s good news and bad news here. The good news is that we are definitely not alone. Many software projects—including open source projects—have worked on these same issues. There are resources all around us. We've created lists of some of them already.

Bitergia dashboards

Open source projects on GitHub

Project-run dashboards

So what should we track?

There's a lot to consider when we think about the health of the Drupal project. We've just begun this conversation and want to make sure you're part of it from the beginning. So, now we turn things over to you.

Which metrics are indicative of Drupal project health? Share your ideas in the comments section. We'll include your feedback in a document we'll share with you.

1 We borrowed these concepts from Product Lifecycle Management.

Flickr photo: Thomas Haynie

Original Article: