Speed up Drupal 7 - MongoDB Field Storage
This weekend, I started a project to standardize on a LAMP and Drupal stack that migrates data as fast as possible. Cyrve's customers tend to have large data sets (e.g. Examiner.com, The Economist, Martha Stewart, World Economic Forum, ...), so insertion speed is crucial. Early throughput observations ...
Environment
Throughput
My Laptop. Untuned Mysql.
900 nodes/minute
My Laptop. MongoDB field storage.
4800 nodes/minute
EC2 with RDS hosted MySQL.
1900 nodes/minute
EC2 with RDS hosted MySQL and MongoDB field storage.
5300 nodes/minute
As you can see, adding mongoDB field storage boosted my laptop by 5x and boosted ECS+RDS by almost 3x. Wow! This is consistent with improvements we experienced with Examiner.com.
Dataset
The dataset I migrated can be found in the migrate_example_baseball module. Here you will find migrate module classes which import a box score from every Major League Baseball game from 2000-2009. All the little bits of the box score (e.g. attendance, batting orders, winning pitcher, etc.) are saved in Fields so this dataset exercises field storage a lot. I think this is typical of larger D7 sites. This dataset should be useful to lots of Drupal benchmarking projects.
Environments
All imports were executed using migrate-import drush command. Like all drush commands, the web server plays no part at all. This is all CLI PHP 5.3, MySQL 5.1, and MongoDB 1.8.
RDS is Amazon's hosted MySQL offerring. I chose a Large instance for all tests. I'm not sure how Amazon configures these, but i would hope that they consider the write-heavy usage scenario is considered.
Caveat: These are not formal benchmarks. I did not try to optimize these environments.
Summary
We already know that MongoDB field storage massively speeds up web page views (i.e. reads). I'm seeing here that it massively speeds up writes as well. Please consider using MongoDB for your large Drupal projects. MongoDb compatibility on its own is a compelling reason to choose Drupal 7 over Drupal 6. Learn more about MongoDB and Drupal from this Drupalcon session.