Certifications in software make me sneeze. Or roll my eyes. Or shrug. Yes, I'm a skeptic of certifications, and leery of motives of people pushing them. To me, certifications are a way to make money not from clients but from peers. It's like a tax. Do not pass Go. Do not collect $200. Instead, pay thousands to some firm so you can get that seal of approval. And in the end, does it mean anything?

And yet there is this obsession with measuring people. It's a way of gatekeeping, of creating scarcity, of one-upsmanship. I don't think such things are a measure of quality. They're typically market tools employed to help one group of people compete over another group. It's part of a "there oughta be a law" approach to life.

Goodness knows there are a lot of people out there who have no business building websites, yet do. Rare is the experienced Drupal developer/themer/site-builder who hasn't been shown some unsightly mess of an implementation that hardly works, if it does at all, with unmaintainable hacked code and untold violations of best practices. But color me an unbeliever when it comes to the salvation offered by some certification system. Certifications don't measure the things that are crucial to effectiveness. They don't measure ability to solve problems. They don't measure those intangible qualities that make people worth working with.

And they can't measure the unknown. Certifications look backwards, and tend to reduce the beautiful and complex to the dry and limited. Are certifications what we want? Really?

Is "Rockstar" measured in inches or decibels?

Certified to Rock attempts to bypass certifications by disrupting them using a mysterious automagical formula. The Certified to Rock algorithm is secret. That's purportedly not a flaw but a feature — presumably because if the algorithms were known, they could be gamed. The blithe response is "if you continue to rock it, your score will increase."

Of course, all this depends upon what you consider a "rockstar". Forget Ozzy and Rod and Moon and Mr Mojorisin. Unlike old-school rock stars, who burned guitars, wrecked hotel rooms, drove cars into swimming pools, bit heads off of doves and passed out on stage, and were paid millions for it, Drupal "rockstars" participate and engage in more socially constructive ways. Yes, baby boomers, the rockstar is the new square. But is your notion in alignment with the notions behind the secret CTR algorithm? Who can say?

I confess that when I first saw Certified to Rock, I thought it was a joke — a kind of frivolous, er, let's say extraordinariness meter for geeks to see how they measured up — "frivolous" because it claims to distill what is ultimately so vague and diverse and multidisciplinary and, well, human that applying a single scale is like comparing apples and cinnamon rolls. And machines are not good at vague and human.

This challenge is not unique to Drupal. Just last week this challenge of defining and measuring qualifications arose at a data science unconference:

One of the best sessions I attended focused on issues related to teaching data science, which inevitably led to a discussion on the skills needed to be a fully competent data scientist.

As I have said before, I think the term “data science” is a bit of a misnomer, but I was very hopeful after this discussion; mostly because of the utter lack of agreement on what a curriculum on this subject would look like. The difficulty in defining these skills is that the split between substance and methodology is ambiguous, and as such it is unclear how to distinguish among hackers, statisticians, subject matter experts, their overlaps and where data science fits.

What is clear, however, is that one needs to learn a lot as they aspire to become a fully competent data scientist. Unfortunately, simply enumerating texts and tutorials does not untangle the knots.

[The author, Drew Conway, offers up a Venn diagram of his own in an attempt to illustrate the complexity of the challenge. If you're interested in data science, data visualization or other big data challenges, the linked posts above are highly recommended.]

Today The New York Times had an article on the challenge of computers understanding humans:

Give a computer a task that can be crisply defined — win at chess, predict the weather — and the machine bests humans nearly every time. Yet when problems are nuanced or ambiguous, or require combining varied sources of information, computers are no match for human intelligence.

Few challenges in computing loom larger than unraveling semantics, understanding the meaning of language. One reason is that the meaning of words and phrases hinges not only on their context, but also on background knowledge that humans learn over years, day after day.

Knowing that CTR is likely not even a mapreduce kind of app, more of a simple query kind of app, one can only ponder its validity in the wild and wooly world of open source interaction development. But the concept has stuck around. And people are starting to cite CTR measurement as a gating criterion for hiring consideration. So there it is. And by that measure, it's worth a closer look.

How does Certified to Rock square that semi-amorphous blob that is a Drupal individual?

The validity of something like CTR is impossible to check, except by anecdote. Like Diebold voting machines, you cannot review the code. Unlike in Drupal, or any open source software, there's no real accountability through openness. You just have to trust it, trust the people behind it, and hope their notions of "rockstar" make some sort of objective sense, their ideas of how to measure those attributes are sound, the metrics they choose to incorporate actually have relevance, the algorithmic factors and quotients employed are properly balanced, and the people behind the curtain stick to high values, integrity and selfless standards despite the obvious and apparent temptations to tip the scales or otherwise succumb to the conflicts of interest inherent in building a system that rates not only your peers but your business competitors.

Even ruling out the last part — and knowing and having worked with Greg, Ezra and Ben, I have no reason to doubt their sincerity or integrity with regard to CTR — that still leaves a lot of unknowns, a lot of uncertainties, a lot of fuzz out of which a single scale of numbers appears — numbers that, despite their vague provenance, are being embraced by some as qualifiers for hire.

So how does it work? The site itself lays out some general parameters:

• It must be easy to automate. There are about 500,000 people on drupal.org and more every day. We can't add something to the system that is going to require much manual work. Certainly nothing that requires us to manually do a task more than a few dozen times. So, we could make a list of everyone who organized a DrupalCon and have the algorithm system use that because that's only a few dozen people (we don't currently do that and may never). But we aren't going to go through github, launchpad, and gitorious and associate people's ID on those systems with their drupal.org user ID and then have the system give credit to people who send lots of e-mails. That's not scalable. Pro tip: keep your work on drupal.org itself - if you don't like the way it looks/functions/whatever then help the redesign.
• It must not encourage anything that is harmful to the community. This is somewhat tricky. If, for example, the system gave points to people who have a lot of projects on drupal.org then that would encourage the creation of lots of projects including a lot of really bad ones. That makes it harder for new users to find projects. Which is really bad. So, we don't use any metrics like that!
• It must be balanced. One of the things we're really trying hard to do is measure skill and knowledge of Drupal in a way that is fair to people with different skill sets. Someone who is an awesome site builder (can't code much, can't design, but really can choose modules and configure them) should be on equal footing with someone who can design or who can code. This is...hard. In particular it is hard to measure the skill/contributions of designers/ux/ia people and site builders. So, if you have ideas on metrics that measure their skills, please share those ideas!

This last point is the hitch. It's the part that gets into the fuzzy logic of AI. How indeed do you compare apples with cinnamon rolls on a single scale? First thing, it seems, is to simply define the scope of what in fact is being measured. What makes a rockin' Drupalist (for lack of a better word)? Programming ability counts, but not if you don't appreciate the foundations of open source. Database fu is great, but is limited if you can't generate decent markup. Theming talent is fabulous, but it's only so much help if you can't build the site to theme. And none of this touches on the intangibles like professionalism, organization, collaborative skills, communication ability, sensitivity to project goals (vs nifty solutions), general mental health, etc. that are critical to effectiveness in a Drupal project.

And yet the machine measures something. What could it be?

Some objective metrics that might be incorporated into a certification algorithm for Drupal

Here are a few ideas (total guesswork on my part), any one of which would be of little use but could possibly be combined into some formula to measure a "rockstar" quotient. Note: These are in no particular order.

Top-Level Metrics

Simple numerical measurements readily at hand.

• Drupal UID [lower is better] — Some assumption of some degree of familiarity or knowledge could be applied based upon duration of membership in the Drupal community. On the other hand, plenty of people joined the community ages ago but haven't been very active. Or maybe you joined in 2003, tried Drupal and moved to Mambo.
• Drupal.org number of posts [higher is better] — Because participation is good. But maybe you're a help vampire or offer not-very-helpful input.
• Drupal.org number of comments [higher is better] — Because engaging in existing conversations shows interest in what the community is doing. But how many comments actually contribute to the conversation?
• Drupal.org number of Issues [higher is better] — Because one can assume that people participating in the issue queues are more engaged in improving Drupal. However, if you're posting a bunch of duplicate issues or build-this-for-me tickets, maybe that's not helping so much.
• Groups.Drupal.org number of posts [higher is better] — Because this shows engagement in more focused subject or regional areas. But if it's a bunch of job spam, who cares?
• Groups.Drupal.org number of comments [higher is better] — Because commenting on existing discussions shows engagement with the community. On the other hand, chatty comments are common on g.d.o and do they make the "rockstar"?
• Drupal.org number of documentation posts [higher is better] — Because if you can teach it, you probably know something. That is, if you do know something and aren't just adding cruft.
• Drupal.org number of documentation revisions [higher is better] — Because making documentation better reflects some understanding of that Drupal subject. But some revisions are (sometimes incorrect) typo or grammar corrections, which speak to one's writing skills but not one's Drupal-fu.
• Drupal.org number of Projects [higher is better, to a point] — Because maintaining a module, theme, feature or profile demonstrates a willingness to contribute, which is an important part of being a Drupal "rockstar". On the other hand, let's face it, there's a lot of crap contributed to CVS.
• Drupal.org number of patches [higher is better] — Because if you're offering patches, your nose is not only in code, it's sniffing out actual solutions to make things better (hopefully). But if your patches are no good, or get ignored because someone can't be bothered, then what does number of patches prove?
• Drupal.org number of core patches [higher is better] — Because if you're working in Drupal core, you're paying attention to the heart of things. There's no knocking this (Drupal core developers are my heroes), but there's a very great oodles more to Drupal than core code, so many are missed by this measure.
• Drupal.org number of core patches in current and impending releases [higher is better] — Because working in the real world of today and tomorrow probably carries more relevance than working on the real world of five years ago or three years from now. Of course, this doesn't mean they're good patches … and the same concern above applies here.
• Drupal.org number of commits by you [higher is better] — Because your projects are actively maintained, which shows engagement and desire to make things better. On the other hand, split out your projects into a bunch of files and you have more commits, and does that make you better, by definition?
• Drupal.org number of commits by others of your patches [higher is better] — Because if others find your patches helpful, all the better. Undoubtedly this is cool, but narrowness of applicability is an issue.
• Recency of all of the above [higher is better] — Because Drupal knowledge can grow stale fairly quickly. On the other hand, if you're really new to this, odds are you have a lot still to learn. (Don't we all!)

Reductive and Social Metrics

These are metrics that would need to be gleaned from some data mining more involved than your basic query. Quantifying these things is what is known as "hard to do":

• Groups.Drupal.org up-ratings of posts [higher is better] — What do others think of your posts? Probably only of low value, as the more pointed remarks generally get the upratings, and this may not reflect any particularly deep Drupal knowledge or engagement at all.
• Drupal.org and Groups.Drupal.org mentions of "thanks" and similar words with your username [higher the better] — This would take some Big Data-type parsing, but could yield some helpful social metrics.
• Drupal.org and Groups.Drupal.org "+1" to posts and patches — This could be difficult to figure, though, as sometimes the +1 is for someone else's comment.
• IRC "karma" points — Although this skews very heavily to the in-crowd, the inner circle talking to the inner circle, as many, if not most, people on IRC don't know about "username++", especially in #drupal-support and #drupal-themes.
• Twitter tweets about Drupal — Because you are probably working with it if you're tweeting about it.
• Blog posts about Drupal — Same deal.

Those Intangibles that Defy Measurement

• How effectively the designer leverages affordances provided by the Drupal UI.
• How well the architect matches Drupal solutions to project challenges.
• How appropriately the themer uses clean html and semantic markup.
• How scalable the developer's solution is in a high-demand application.
• How efficiently the project manager directs resources towards needs in a Drupal project.

And these don't even touch the non-Drupal-specific intangibles that apply in every Drupal project:

• How collaborative a person is.
• How professional a person is.
• How hard-working a person is.
• How responsible a person is.
• How intelligent a person is.
• A person's work ethic.
• A person's attention to detail.
• A person's adaptivity to change.
• A person's poise under pressure.
• A person's values and empathy towards others.
• Honesty.
• Punctuality.
• Reliability.
• Focus.
• Discipline.
• Insight.
• Critical thinking.

And so on.

How many is a flower?

I don't know the answer to that. Is orange greater than salty? Is platinum better than mitochondria? Yes. No. Maybe.

And?

topics: 

Drupal,
community,
web development,
Certified to Rock,
certifications.