Need to store up some items and process them en masse in a cron job? That’s a perfect use case for the Queue API (for Drupal 7+) or the drupal_queue module (the backport for Drupal 6).

Concrete example: One of our consulting clients was using a synchronous webservice for their retail store locator. We needed to create a form where store owners could change whether they’re listed in the store locator. When a store owner submits the form, should she have to wait around while our site sends her change to the store locator webservice? We didn’t think so. So we decided to use Drupal’s Queue API, which lets us save up all the store owners’ changes and send them to the webservice in a batch.

Here’s a simple example of adding items to a queue in a form submit handler and processing them in the next cron run.

A simple example

First, we tell Drupal that we’ll be using a queue called example_queue and that, when cron runs, the _example_print_item function should be called for each item in example_queue.

/*
* Implements hook_cron_queue_info().
*/
function example_cron_queue_info() {
$queues['example_queue'] = array(
'worker callback' => '_example_print_item',
);
return $queues;
}
 
function _example_print_item($dequeued_item) {
drupal_set_message(t('Dequeued: %item', array('%item' => $dequeued_item)));
}

Now, we just have to create a form that lets the user put items into the queue.

/*
* Implements hook_menu().
*/
function example_menu() {
$items['enqueue'] = array(
'title' => t('Enqueue'),
'type' => MENU_CALLBACK,
'page callback' => 'drupal_get_form',
'page arguments' => array('_example_form'),
'access callback' => TRUE,
);
return $items;
}
 
function _example_form($form_state) {
$form['item_to_enqueue'] = array(
'#type' => 'textfield',
'#title' => t('Item to enqueue'),
'#default_value' => date('H:i:s'),
'#required' => TRUE,
);
$form['submit'] = array(
'#type' => 'submit',
'#value' => t('Submit'),
);
return $form;
}
 
function _example_form_submit($form, &$form_state) {
$item_to_enqueue = $form_state['values']['item_to_enqueue'];
$queue = DrupalQueue::get('example_queue'); // Drupal 7
//$queue = drupal_queue_get('example_queue'); // Drupal 6
$queue->createQueue();
$queue->createItem($item_to_enqueue);
}

A complex example

For an example that exercises more of the features of queues, check out the excellent Examples For Developers module.

A hook that gets executed just before queues are processed

For the store locator webservice I mentioned earlier, each call to the worker callback would send a message to the webservice with one store owner’s change. But before sending these messages, we needed to log in to the webservice.

The Queue API provided just the hook we needed: hook_cron_queue_info_alter(). It gets called during a cron run, just before Drupal starts calling the worker callbacks on the enqueued items.

An enqueued item may get processed more than once

From the Queue API documentation:

There are two kinds of queue backends available: reliable, which preserves the order of messages and guarantees that every item will be executed at least once. The non-reliable kind only does a best effort to preserve order in messages and to execute them at least once but there is a small chance that some items get lost… regardless of the queue being reliable or not, the processing code should be aware that an item might be handed over for processing more than once (because the processing code might time out before it finishes).

Unless you’ve gone out of your way to change the type of queue your system is using (by setting the variable queue_class_$name, where $name is the name of your custom queue implementation), you don’t have to worry about the first part; the default queue is reliable. But you do need to be aware that your worker callback might get called more than once per item.