Topic:

• Drupal
• education

I've been burning to tell you about an educational experiment. The challenge of the experiment was to deal with a problem of reduced support (teachers) and increase number of students (over 300). The solution: add some community spirit. I've been using my Drupal experience to guide this challenge and did some development along the way. We extended the issue queue to work with peer-evaluations and make learning part of the solution instead of the problem.

In this blog I will only give a sneak preview about the Drupal interface. With Prof. K. Lombaerts, I will work on an academic paper in an educational journal to elaborate the strategic policy for self-regulative education. The educational experiment was done with Prof. F. Plastria course Software for Management, which is an advanced spreadsheet course that Master-level business students have to learn to understand how they can model, analyze and simulate datasets to deal with uncertainty. The magnitude of this experiment was huge. I've spend over 650 hours spread over 4 months. Let me throw some more numbers at you to grasp the magnitude of the project.

The experiment involves 410 subscribed students, where 330 actually posting something, however 30 of them barely did any work, so lets say 300 students. In total, these students posted 4276 nodes and 7980 comments. Two content types were more important then others: issues and evaluations. In the end 767 issues across 209 projects got evaluated by 1709 evaluations created by students and 223 evaluations created by two teachers. So each issue had more then 2 evaluations and average 2,5 issues per student got created. Considering that a student would spend around 10 hours for an issue and around 3 hours for an evaluation, I'm hoping your appreciate the magnitude of the educational experiment.

The setup

I've been working with this Drupal site for master-level education ever since 2006. Each year never more then 40 people, so this one experiment was as intensive as the past 6 years together. As learned from previous experimentation, we turned the classroom exercises and the homework inside out. The prior classroom exercises got replaced by videos (see http://www.youtube.com/mixelkiemen) and projects were brought to the classroom. However the amount of students wouldn't allow for much direct contact. The classroom was indented to be a workspace for students to collaborate by informal interaction. Most actual activities happened via formal interaction with the issue-queue and other structures of the course site.

A project got broken down into issues; three categories existed (model, analysis and simulations) that had three possible levels (basic, normal and advanced). Each issue would upload an Excel file, but the spreadsheet is only the base, it’s the data modeling, sensitivity analyzing and (Monte Carlo) simulations that are truly important. Each issue required at least two evaluations by another student. Of course you're not sure if an evaluation is correct, therefore a meta-evaluation on each evaluation was needed.

While I first had to bootstrap the course and so focus on developing projects, I soon find myself shifting to developing evaluations, meta-evaluation and lastly supervise my meta-evaluation team. In the end we had 60 students creating meta-evaluations. Considering that evaluators corrected some of the mistakes made by the first student and that meta-evaluators corrected mistakes in the evaluation, we reduced the required feedback significantly. Thus the teacher's focus was to correct the quality and elaborate the exceptions.

Now the real data-analysis is only starting, but we have learned how to significantly improve this process (so we think) and already consider the innovation experiment worth the effort.

The simplified conclusion is that "gamification" mediates autonomy and collaboration. Still this is oversimplified; some specific problems have to be addressed. The gamification does allowed an instantly calculated of the score (see the calculation example http://mosi.vub.ac.be/webdev/?q=node/2170), which lead to faster feedback for students, but also a reduction of the administration.

The development

From a technical point the site was pretty simple. Considering technical limits at our university I had to work with Drupal 6. I've first hacked the project module, for some minor changes, like changing "category" to "level". Then I've used 5-stars-CCK to create the evaluation type and some basic stuff like node reference. Still as time passed I would find myself programming more, in the end I've created three interfaces, one for the students, one for the evaluators and one for the teachers. The student interface was the first we developed. It was to give students feedback on their work. The total score would show the total point earned on issues and evaluations and we created some simple system of bonus points. This would lead to a score on 20:

The total score is an abstraction. On the one hand students could earn point for each of the three types (model, sensitivity analysis and MC simulations), leading to an issue score table (on 10):

Similarly a table for evaluations existed. For each issue a student uploaded 2 evaluations were needed, leading to individual criteria for all students. Students could do more evaluations then needed, which would increase their score. For example SvenM needs 8 evaluations and did 10, so the system would look at the average of the 8 best evaluations, which is 0.6 higher then his normal average. At the moment 1 evaluation is still pending, meaning it did not get meta-evaluated. Students would get 0.1 bonus-points for each meta-evaluation they do. Both the meta-points or any penalties would get calculated into the evaluation score.

By the end of the semester 60 students got selected after showing some good evaluation skills. They became part of the meta-evaluators team, in next table you see students with more then 20 meta-evaluations. Considering all the pressure and work, we got surprised how much good meta-evaluation these students did:

Still we got equally surprised of the continuing spamming of some of the worst students. It seems as the classroom setting would hardly have any influence on community dynamics, with the exception that some of my meta-evaluations got phoned up by bad students and became threaten …. not something we had anticipated at all !

With the evaluations in full progress we saw the need to create different interfaces. The meta-evaluators would be in constant email traffic with me and other students mailed too. During the evaluation period (January 2012) I would get to 477 emails that need instant reply, that is over 15 mails a day! Next to my daly quotum of 20 issues and the programming, this lead to very long working hours. One of the conflicts led to creating a table for meta-evaluators to get a holistic view on some of the students' behavior:

We also created some extra views for students so they could get a selective list with all issues that needed review or all issues that required dummy evaluations (issues that only had bad/poor evaluations).

Finally I also created some interface for the teacher, with some general info, but also a list of meta-evaluated issues and a list of candidate evaluators:

The most work was all the calculations and the terrible complicated mysql queries to create this. This probably could have been programed easier, but I was on time pressure and so created some ugly solutions that worked. I didn't upload the code; it could be implemented a lot better. It was just to run the experiment. The goal is to use this experience to design some good software architectures. Still I'm only interested in designing, not build them. I wish there was a way to collaborate with some developers on this or maybe some shops as an R&D project, however I lost my fait that such a thing will happen … let me elaborate.

Drupal & Academics, closure, finally.

Drupal is fast, compared to any development. Academic development is often slow, but can be significant even for fast projects like Drupal. With Drupalcon Brussels (2006) I understood the challenge for education and got the opportunity to apply this almost instantly While I took on the challenge only days after Drupalcon, it took me 5 year to run experiments and I've only been able to express it properly some months ago: see my presentation @ DrupalGovDays.

Still, this has not been relevant for Drupal, none of my students became part of the community as far as I can tell. In contract Summer of Code (SOC) students have been vital to Drupal's core. I've been trying to have a debate on getting from SOC to something more sustainable, but the discussion has never lead to strong plans. Today I understand the magnitude of such efforts and now I understand this can't be solved easily. It will be doable, but a big investor would be needed.

In Drupalcon Szeged (2008), I've tried to organize a BoF on research, but it got mixed up with another BoF lead to no follow up. In Drupalcon Copenhagen (2010) & Chicago (2011) I did several interviews to get an overview of founding companies that pioneered Drupal's business ecosystem. This fit my research on "self-organizing innovation", which I relate to open innovation, as I'm elaborating in my papers. While I made a first working paper in 2011 and have been publishing in some conference proceeeding, the actual academic paper for a journal is still work in progress. In fact it has been split up as a case for two papers, the first journal publication is plan to appear in 2013. So it is better then 5 years that was needed for the first experiment, but still 3 years after the event took place. Inside the Drupal community 3 years is long, so I see the bought in peoples eyes when they talk to me. In all the years fighting for this mixture of R&D development, I'm loosing fait. I will finish the papers and they probably will have no impact for the Drupal community, just as 5 year teaching a Drupal course had zero impact. I'm getting more and more convinced my energy is directed wrongly.

For London I did more effort to organize an open space "DrupalEdu", while the open space was successful, the follow up was not. As expressed in my last blog, I'm tired of tilting at windmills. I see a problem of "architectural failure", which is ironically, as it is something that shows up in my radical innovation research all the time. I feel lost, burnout and becoming an outcast in respect to Drupal community. I've made some friends along the way, but I've seen good friend leave too. I've got plans to go to Munich, but I'm not exactly sure why. One thing is sure, I don't intent to push anything without serious support.

About serious support, I'm proud to announce that the "Global Brain Institute" (GBI) got founded. The first target of the GBI, which we call the first pillar, is a mathematical model. 4 PhD students have just been assigned to work on this research for the next 5 years. This first pillar of the GBI has little to do with my own interest on radical innovation, but with my advisor we designed a second pillar related to self-regulating education. This second pillar is important to us, but we don't have any investment for it yet. We are allowed to attract co-financing for the GBI.

My first focus during 2012 will be to finish my PhD, but if I can find any opportunity for the second GBI pillar, I won't be neglecting it. In fact there are some discussions going on, but talking is cheap. Research plans are only real once the contract is signed, past disappointments have made me grim and skeptical, they taught me not to get my hopes up high. Patience is a blessing I've still got to learn. Still this month GBI first pillar is founded, so let' s think positive and try to smile to the future.