Eucalyptus, QA

Extracting Info From Euca’s Logs

logstashIntroduction

Throughout my tenure as a Quality Engineer, I have had a love/hate relationship with logs. On one hand, they can be my proof that a problem is occurring and possibly the key to tracking down a fix. On the other hand, they can be an endless stream of seemingly unintelligible information. In debugging a distributed system, such as Eucalyptus, logging can be your only hope in tracing down issues with operations that require the coordination of many components.

Logs are generally presented to users by applications as flat text files that rotate their contents over time in order to bound the amount of space they will take up on the filesystem. Gathering information from these files often involves terminal windows, tail, less, and timestamp correlation. The process of manually aggregating, analyzing and correlating logs can be extremely taxing on the eyes and brain. Having a centralized logging mechanism is a great leap forward in streamlining the debug process but still leaves flat text files around for system administrators or testers to analyze for valuable information.

A month or so ago I sought out to reinvigorate my relationship with logs by making them sexy again. I looked around at the various open source and proprietary tools on the market and decided to give Logstash a shot at teaching me something new about Eucalyptus through its logs. The “getting started” links I found on the docs page presented a quick and easy way to see what LogStash could do for my use case, namely ingesting and indexing logs sent from rsyslog. Once I got some logs to appear in the ElasticSearch backend, I got a bit giddy as I was now able to search and filter the logs through an API. But alas! I was still looking at text on a freaking black and green screen. BORING! There had to be a better way to visualize this data.

I looked around a bit and found Kibana. This beautiful frontend to ElasticSearch gives you a simple and clean interface for creating/saving dashboards that reflect interesting information from your logs. Within minutes of installing Kibana, I had a personalized dashboard setup that was showing me the following statistics from my Eucalyptus install that was undergoing a stress test:

  • Instances run
  • Instances terminated
  • Volumes created
  • Volumes deleted

I had proven that there was value in using Logstash and it was not complicated to setup or use. I then began to use other dashboards, filters, and search terms to look for anomalous patterns in the log messages. This type of analysis resulted in a couple of issues being opened that I would not have found looking at one screen of text at a time.

Below I will outline the steps to begin your own Logstash journey with Eucalyptus or any other system/application that logs to a filesystem on a Linux box.

Installation

Installing Logstash

  1. Install packages
    • On Ubuntu: 
      apt-get install default-jre git apache2 ntp
    • On CentOS:
      yum install java-1.7.0-openjdk.x86_64 git httpd ntp
  2. Set proper timezone
    1. Ubuntu
    2. CentOS
  3. Download Logstash
    • wget https://logstash.objects.dreamhost.com/release/logstash-1.1.13-flatjar.jar -O logstash.jar
  4. Create LogStash config file for rsyslog input. Create and edit a file named logstash.conf
    • input {  syslog {    type => syslog    port => 5544  }}
      output {  elasticsearch { embedded => true } }
  5. Run logstash JAR
    • nohup java -jar logstash.jar agent -f logstash.conf &
  6. Configure rsyslog on Eucalyptus components by adding the following to the /etc/rsyslog.conf file and replacing <your-logstash-ip>
    • $ModLoad imfile   # Load the imfile input module$ModLoad imklog   # for reading kernel log messages
      $ModLoad imuxsock # for reading local syslog messages
      $InputFileName /var/log/eucalyptus/cloud-output.log
      $InputFileTag clc-sc-log:
      $InputFileStateFile clc-sc-log
      $InputRunFileMonitor
      $InputFileName /var/log/eucalyptus/cc.log
      $InputFileTag cc-log:
      $InputFileStateFile cc-log
      $InputRunFileMonitor
      *.* @@<your-logstash-ip>:5544
  7. Restart rsyslog
    • service rsyslog restart

Installing Kibana 3

  1. Clone the repository from GitHub
    • git clone https://github.com/elasticsearch/kibana.git
  2. Edit the kibana/config.js file and set the elasticsearch line to:
    • elasticsearch:    "http://<your-logstash-public-ip>:9200", 
  3. Copy the Kibana repository to your web server directory
    • CentOS:
      mv kibana/* /var/www/html/; service httpd start
    • Ubuntu:
      mv kibana/* /var/www/

Point your browser to http://<your-logstash-public-ip&gt; and you should be presented with the Kibana interface. Kibana is not specifically a frontend for Logstash but rather a frontend to any ElasticSearch installation. Kibana does provide a default Logstash dashboard as a starting point for you customizations:  http://<your-logstash-public-ip>/index.html#/dashboard/file/logstash.json

Standard

3 thoughts on “Extracting Info From Euca’s Logs

  1. Tim says:

    Is it possible you could share your filters for Euca? Using your steps I got it running. It’s a bit outdated on the links and wouldn’t work without downloading the latest versions. I also had to separately google and load Elasticsearch since that was not included in a standard Centos 6.5 version. Thanks for the great stuff!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s