24 Jun

Brian Gilbert

At DrupalCon Sydney where I was the Community track chair, for the first time at a DrupalCon both Stuart and I had offered to help as mentors at the post conference sprint. We felt in a relatively good position to do this as we’ve been running a mentoring event locally in Melbourne nearly every month since some time in 2010, which is another story in itself.

This naturally led on to us being mentors at DrupalCon Portland. During the mentor BOF’s while mostly listening, I was already thinking how can I make things better and the thing that stuck out to me were the common issues that often came up while getting a first time sprinters set up with a local development stack:

• Bandwidth usage
• Different stacks (MAMP, WAMP, XAMPP, etc)
• Mentors having to support a stack they’re not familiar with
• Time of setup (some users would take nearly an hour to setup)

Around the same time Stuart and I had been playing around with BitTorrent Sync (BTSync), which at that time was pretty new, but in my mind a promising potential solution to the bandwidth component of the problem.

I also thought that Acquia Dev Desktop (ADD) may be a better option for setting up a local stack than using any of the other *AMP stacks listed above. While also simplifying the support task for mentors that are essentially volunteering their time.

Eventually I started to talk to some of the more seasoned mentors about my ideas and there was some interest in my thoughts on BTSync, but a lot of kickback on using ADD due to previous sprints where various people had been burned trying to use ADD with Drupal 8 in the past.

As luck would have it one of the mentors that worked at Acquia, scor AKA Stéphane Corlosquet, was at these BOF’s also and we exchanged details. See the relevant issue at: https://www.drupal.org/node/2232043.

Attending what is now known as the First Time Sprinters workshop in Portland I was surprised when it was suggested that attendees all go and download MAMP/WAMP/XAMPP “now”.. It was no wonder the wifi had a history of dying during sprints, and it didn’t disappoint this time going down at exactly this point during the Portland sprints.

Needless to say this was enough motivation for me to implement and prove my thoughts, so I started building what eventually became the Community Tools Installation process.

What were the constraints?

There wasn’t that many, but these were the key things:

• Installation is fast
• Lean on the network
• Doesn't break anything existing on a users machine.

It seems common for people to think that we are putting this forward as the ideal development stack, this is not at all the case, it is simply designed to be the fastest way to get someone up and running with a capable stack for the sprint day.

By using BTSync the majority of the network traffic for the tools is served from within the sprint room itself from the laptops of mentors that have pre-synced it.

BTSync has two levels of access to a shared folder, one that is read/write (only known to me) and one that is read only which is given to everyone else. This means we don’t run the risk of spreading malware that could arise through the use of USB keys shared with approximately 200 people.

BTSync also allows for last minute changes to be made that will sync out to all users, meaning we can respond to issues that arise quickly and not have to re-image a bunch of USB keys just before the workshop.

In the beginning there was a lot of pressure to include MAMP and other tools in the package, but I was eventually able to work with Acquia via scor to improve the ADD2 Beta such that it was the clear winner for this specific use case. Related issue: https://www.drupal.org/node/2232043.

ADD2 installs it’s webserver and database using ports for these services that are to my knowledge unique, meaning that it will not conflict with any existing web stack a user has set up.

ADD2’s supplied configurations (PHP/MySQL) also work out of the box with Drupal 8, something that often needed changing with other stacks.

Why this collection of tools?

The components were primarily chosen based on relevance to the typical activities performed by a first time sprinter during the sprint day, these are:

• Use IRC (Limechat / Hexchat)
• Edit code (SublimeText)
• Use Git to apply and create patches (Git)
• Run Drupal 8 (Acquia Dev Desktop 2)
• Install Drupal 8 (Drupal Core — dev branch)

Initially the tools came with a PDF that walked you through each of the installation steps, pretty quickly I saw that people weren't reading this and making mistakes in the installation, so fairly promptly I started scripting as much of the installation as possible to reduce the chance of user error and also reduced the installation time. Related issue: https://www.drupal.org/node/2294267.

The OSX scripting was easy but It was fun having to learn how to code a .BAT file after not having used windows for several years.

At this stage the only manual step of the installation apart from selecting which parts you want to install is adding the site in ADD which has substantially reduced the size of the PDF that nobody reads ;)

The script prompts to see if you want to install an IRC client and a code editor, it then checks if you have git installed or not and prompts to install and configure git to d.o standards.

To save bandwidth Drupal 8 core is included in the tools package that is synced, extracted during the installation and then a git pull is executed to only grab recent changes saving again on bandwidth.

At this stage I’m very confident this is as lean as it can get without creating a customised install package specific to each operating system.

How has it performed?

After testing it at local events the concept was proven, the first real test at scale at DrupalCon Austin, about 50 people out of 200 had issues with it syncing but I considered it a success.. and the network stayed up!

We used it again in DrupalCon Amsterdam with closer to 300 attendees where I don’t think we had as many issues with people getting it synced but did still have to use USB drives for some people.

And I used it at DrupalSouth Melbourne earlier this year with approximately 100 people where it synced perhaps the fastest ever and we didn't have to use any USB thumb drives.

It was used at DrupalCon LA, I'm chasing up some statistics on how it went and will update asap.

What could be improved?

It’s fairly clear that there are some limitations built into BTSync, and that the venue network setup can sometimes cause issues. I am constantly on the lookout for a better transport method (ideally looking for something open source instead of BTSync or for the Sync team to provide a free pro account so that we can share these secrets with unlimited users)

I think having a web server at the venue that could serve the tools would provide the most added benefit to the existing setup, while removing the weakest remaining link (BTSync) and also opening up the possibility of using tools that are larger in file size.

Linux doesn’t have ADD so the current experience is quite different for Linux users, and in my not consistent enough. To some degree we expect these users to already know how to set up a stack which isn’t the best assumption to make.

Statistics

Tools download file sizes:

• Linux         510MB
• OS X         247.5MB
• Windows  290.3MB

External Bandwidth usage during installation:

On OSX Git is installed via download of the command line tools.. ~120MB
For OSX and the rest the only the Git pull to update core ~9MB or less

Installation time:

• Linux          ~5 minutes
• OS X           ~4 to 7 minutes depending on if you already had git installed or not
• Windows     ~5 minutes

Where to from here?

There are several community members that are pushing for vagrant to be for the stack part of this toolset, I think that it will go a long way to resolving the inconsistencies that we currently see with Linux setups. Talking about it in the office we all think it needs a pretty control panel to make starting and stopping the vagrant instance easier for as wide a group of end users as possible. But there are some considerations that still need to be factored in. Related issues: https://www.drupal.org/node/2233509 and https://www.drupal.org/node/2232049.

This is a recount of events spanning several years, I've done my best to be as accurate as possible get in touch and help me clarify things further.

drupal planetdrupal 8