We are crazy about graphs and statistics. No wonder we also want to get a feel for the health of our Drupal sites in some way. On our video wall, one screen is dedicated to showing us errors per server to get a better understanding for how good we are doing. You cannot reach zero here, but you see if something fails miserably when things go over 9,000!

We can use our logging information for several things - visualizing error messages is just one use case. As an example, we store the generated data a bit longer to audit some processes, for example, content modification or deletion which is logged by the content module. However, let's jump in!

Architecture

Our setup is currently pretty simple. The so called Elasticsearch ELK Stack (ELK stands for Elasticsearch Logstash Kibana) provides many more possibilities to setup for scalability. For our current size and needs we are happy with the way we set it up. It is straight forward and empowers us to make use of the data that is otherwise stored in the Drupal databases mostly unnoticed.

Drupal - Syslog Module

We make use of the Syslog functionality within Drupal to ship the Syslogs to our Local 7 endpoint of Syslog. From there, the messages are passed to Logstash.

Enabling Syslog is a straightforward thing:

drush en syslogdrush vset syslog_identity yoursite_comdrush vset syslog_facility 184drush dis dblog

The remaining settings are left default, but you can change the information that is logged to your needs.

Logstash

Logstash, written by Jordan Sissel, is the Swiss Army knife in our setup. You can think of it as Multiple Input - Multiple Output System. It features many In and Output Types (check the documentation section of the Logstash website for more information). We use the Syslog listener of Logstash and feed this data into our Elasticsearch instance. We create a new index every day to ease the process with retaining the logfiles. The current setup does not need further separation to this, Logstash allows us to split up the messages passed along from Drupal into separate searchable fields. For example, we split the site name, username, module, severity into separated fields.

Elasticsearch

All log entries are passed on from Logstash to Elasticsearch for indexing, search and storage. Elasticsearch is a quite young but powerful tool. It handles sharding and replication over many nodes with ease and has fantastic performance in searching.

Kibana

Kibana is the frontend to Elasticsearch. It allows you to query the data stored within Elasticsearch and displays this information in graphs, diagrams tailored to your needs. We have several graphs we use daily. For example, we show all exceptions thrown on our video wall to see when something goes down. A sudden change in quantity of the errors gives us a hint to look into it. We also go trough the errors after deployments to ensure no fatal exceptions happen and are able to fix things that wouldn't be noticed otherwise.

Want to hear more?

Got curious to see what you can achieve with those tools? I'll is speaking at Campus Berne on 22. April 2014 - You can register yourself on TechUp.

Stay in the loop – join our newsletter!