Eucalyptus, QA

EucaLoader: Load Testing Your Eucalyptus Cloud

Locust-full-page

Introduction

After provisioning a cloud that will be used by many users, it is best practice to do load or burn in testing to ensure that it meets your stability and scale requirements. These activities can be performed manually by running commands to run many  instances or create many volumes for example. In order to perform sustained long term tests it is beneficial to have an automated tool that will not only perform the test actions but also allow you to analyze and interpret the results in a simple way.

Background

Over the last year, I have been working with Locust to provide a load testing framework for Eucalyptus clouds. Locust is generally used for load testing web pages but allows for customizable clients which allowed me to hook in our Eutester library in order to generate load. Once I had created my client, I was able to create Locust “tasks” that map to activities on the cloud. Tasks are user interactions like creating a bucket or deleting a volume. Once the tasks were defined I was able to compose them into user profiles that define which types of actions each simulated user will be able to run as well as weighting their probability so that the load can most closely approximate a real world use case. In order to make the deployment of EucaLoader as simple as possible, I have baked the entire deployment into a CloudFormation template. This means that once you have the basics of your deployment done, you can start stressing your cloud and analyzing the results with minimal effort.

Using EucaLoader

Prerequisites

In order to use EucaLoader you will first need to load up an Ubuntu Trusty image into your cloud as follows:

# wget https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
# euca-install-image -i trusty-server-cloudimg-amd64-disk1.raw -n trusty -r x86_64 -b trusty --virt hvm

We will also need to clone the EucaLoader repository and install its dependencies:

# git clone https://github.com/viglesiasce/euca-loader
# pip install troposphere

Next we will upload credentials for a test account to our objectstore so that our loader can pull them down for Eutester to use:

# euare-accountcreate loader
# euca_conf --get-credentials  loader.zip --cred-account loader
# s3cmd mb s3://loader
# s3cmd put -P loader.zip s3://loader/admin.zip


Launching the stack

Once inside the euca-loader directory we will create our CloudFormation template and then create our stack by passing in the required parameters:

# ./create-locust-cfn-template.py > loader.cfn
# euform-create-stack --template-f loader.cfn  loader -p KeyName=<your-keypair-name> -p CredentialURL='http://<your-user-facing-service-ip>:8773/services/objectstorage/loader/admin.zip' -p ImageID=<emi-id-for-trusty> -p InstanceType=m1.large

At this point you should be able to monitor the stack creation with the following commands

# euform-describe-stacks
# euform-describe-stack-events loader

Once the stack shows as CREATE_COMPLETE, the describe stacks command should show outputs which point you to the Locust web portal (WebPortalUrl) and to your Grafana dashboard for monitoring trends (GrafanaURL).


Starting the tests

In order to start your user simulation, point your web browser to the the WebPortalUrl as defined by the describe stacks output. Once there you can enter the amount of users you’d like to simulate as well as how quickly those users should “hatch”.

Locust-start-test

Once you’ve started the test, the statistics for each type of requests will begin to show up in the Locust dashboard.

Locust-test-running


See your results

In order to better visualize the trends in your results, EucaLoader provides a Grafana dashboard that tracks a few of the requests for various metrics. This dashboard is easily customized to your particular test and is meant as a jumping off point.

Locust-dashboard

Standard
Eucalyptus, QA

Testing Riak CS with Eucalyptus

EUCA+RIAK-CS

Introduction

One of the beautiful things about working with IaaS is the disposable nature of instances. If they are not behaving properly due to a bug or have been misconfigured for some reason, instances can be terminated and rebuilt with more ease than debugging a long lived and churned through Linux system. As a quality engineer, this dispensability has become invaluable in testing and developing new tools without needing to baby physical or virtual machines.

One of the projects I have been working on lately is an easy deployment of Riak CS into the cloud in order to quickly and repeatedly test the object storage integration provided by Eucalyptus in the 4.0 release. Riak CS is a scalable and distributed object store that provides an S3 interface for managing objects and buckets.

Before testing the Eucalyptus orchestration of Riak CS (or any tool/backend/service that Euca supports for that matter), it is important to understand the basic activities that Eucalyptus will be performing on behalf of the user. Thankfully, Neil Soman wrote a great blog post about how our Riak CS integration is designed.

In this model  we can see that we require:

  1. A multi-node Riak CS cluster
  2. A load balancer
  3. A machine to run the Eucalyptus Object Storage Gateway (OSG)

This topology is extremely simple to deploy in Eucalyptus 3.4 using our ELB and by using Vagrant to deploy our Riak CS cluster. Here’ s how to get your groove on.

Prerequisites

  1. CentOS 6 image loaded into your cloud
  2. Keypair imported or created in the cloud
  3. Security group authorized for port 8080,8000 and 22
  4. Install Vagrant

Deploy Riak CS

In order to deploy Riak CS in our cloud we will use Vagrant+Chef+Berkshelf as follows:

  1. Install Vagrant plugins using the following commands:
    • vagrant plugin install vagrant-berkshelf
      vagrant plugin install vagrant-omnibus
      vagrant plugin install vagrant-aws
  2. Import the dummy vagrant box necessary to use vagrant-aws:
    • vagrant box add centos dummy.box
  3. Clone the following repository
    • git clone https://github.com/viglesiasce/vagrant-riak-cs-cluster.git
  4. Edit the following items in the Vagrantfile to reflect the pre-requisites above and to point to your target cloud
    • aws.access_key_id
    • aws.secret_access_key
    • aws.keypair_name
    • aws.ami
    • override.ssh.private_key_path
    • aws.security_groups
    • aws.endpoint
  5. Set the number of nodes to deploy at the top of the Vagrantfile:
  6.  Once the cloud options are set start the Vagrant “up” process which will deploy the Riak CS nodes and Stanchion:
    • RIAK_CS_CREATE_ADMIN_USER=1 vagrant up --provider=aws
  7. Once Vagrant is complete, login to the first Riak CS node to get its private hostname:
    • vagrant ssh riak1 -c "curl http://169.254.169.254/latest/meta-data/local-hostname"
  8. Join each node to the first that was deployed. For example, to join the second node to the cluster I would run:
    • vagrant ssh riak2 -c "riak-admin cluster join riak@<riak1-private-hostname>"
      vagrant ssh riak2 -c "riak-admin cluster plan; riak-admin cluster commit"

In order to get your access and secret keys login to http://riak1-public-ip:8000

Load Balance Your Riak CS Cluster

  1. Create an ELB with the following command:
    • eulb-create-lb -z <AZ-of-your-riak-nodes> -l "lb-port=80, protocol=TCP, instance-port=8080,instance-protocol=TCP" RiakCS
  2. The command above will return you the DNS name that you will use as the endpoint for the “objectstorage.s3provider.s3endpoint” property when setting up the OSG. From the sample output below we would use “RiakCS-229524229045.lb.home”
    • DNS_NAME        RiakCS-229524229045.lb.home
  3. Register your Riak CS nodes with that load balancer:
    • eulb-register-instances-with-lb --instances <instance-id-1>,<instance-id-2> RiakCS

You have now successfully deployed a Riak CS cluster. You can stop here if you’d like but the real fun starts when you add IAM, ACL, versioning, multipart upload, and bucket lifecycle support to the mix using the Eucalyptus OSG.

True enthusiasts continue below.

Install and Configure the Eucalyptus OSG Tech Preview

  1. Spin up another CentOS 6 instance in the same security group as used above
  2. Follow the instructions found here to finish  the OSG installation and configuration, remember to use the DNS name returned in step 1 from above as the s3endpoint:
Standard
Eucalyptus, QA

Extracting Info From Euca’s Logs

logstashIntroduction

Throughout my tenure as a Quality Engineer, I have had a love/hate relationship with logs. On one hand, they can be my proof that a problem is occurring and possibly the key to tracking down a fix. On the other hand, they can be an endless stream of seemingly unintelligible information. In debugging a distributed system, such as Eucalyptus, logging can be your only hope in tracing down issues with operations that require the coordination of many components.

Logs are generally presented to users by applications as flat text files that rotate their contents over time in order to bound the amount of space they will take up on the filesystem. Gathering information from these files often involves terminal windows, tail, less, and timestamp correlation. The process of manually aggregating, analyzing and correlating logs can be extremely taxing on the eyes and brain. Having a centralized logging mechanism is a great leap forward in streamlining the debug process but still leaves flat text files around for system administrators or testers to analyze for valuable information.

A month or so ago I sought out to reinvigorate my relationship with logs by making them sexy again. I looked around at the various open source and proprietary tools on the market and decided to give Logstash a shot at teaching me something new about Eucalyptus through its logs. The “getting started” links I found on the docs page presented a quick and easy way to see what LogStash could do for my use case, namely ingesting and indexing logs sent from rsyslog. Once I got some logs to appear in the ElasticSearch backend, I got a bit giddy as I was now able to search and filter the logs through an API. But alas! I was still looking at text on a freaking black and green screen. BORING! There had to be a better way to visualize this data.

I looked around a bit and found Kibana. This beautiful frontend to ElasticSearch gives you a simple and clean interface for creating/saving dashboards that reflect interesting information from your logs. Within minutes of installing Kibana, I had a personalized dashboard setup that was showing me the following statistics from my Eucalyptus install that was undergoing a stress test:

  • Instances run
  • Instances terminated
  • Volumes created
  • Volumes deleted

I had proven that there was value in using Logstash and it was not complicated to setup or use. I then began to use other dashboards, filters, and search terms to look for anomalous patterns in the log messages. This type of analysis resulted in a couple of issues being opened that I would not have found looking at one screen of text at a time.

Below I will outline the steps to begin your own Logstash journey with Eucalyptus or any other system/application that logs to a filesystem on a Linux box.

Installation

Installing Logstash

  1. Install packages
    • On Ubuntu: 
      apt-get install default-jre git apache2 ntp
    • On CentOS:
      yum install java-1.7.0-openjdk.x86_64 git httpd ntp
  2. Set proper timezone
    1. Ubuntu
    2. CentOS
  3. Download Logstash
    • wget https://logstash.objects.dreamhost.com/release/logstash-1.1.13-flatjar.jar -O logstash.jar
  4. Create LogStash config file for rsyslog input. Create and edit a file named logstash.conf
    • input {  syslog {    type => syslog    port => 5544  }}
      output {  elasticsearch { embedded => true } }
  5. Run logstash JAR
    • nohup java -jar logstash.jar agent -f logstash.conf &
  6. Configure rsyslog on Eucalyptus components by adding the following to the /etc/rsyslog.conf file and replacing <your-logstash-ip>
    • $ModLoad imfile   # Load the imfile input module$ModLoad imklog   # for reading kernel log messages
      $ModLoad imuxsock # for reading local syslog messages
      $InputFileName /var/log/eucalyptus/cloud-output.log
      $InputFileTag clc-sc-log:
      $InputFileStateFile clc-sc-log
      $InputRunFileMonitor
      $InputFileName /var/log/eucalyptus/cc.log
      $InputFileTag cc-log:
      $InputFileStateFile cc-log
      $InputRunFileMonitor
      *.* @@<your-logstash-ip>:5544
  7. Restart rsyslog
    • service rsyslog restart

Installing Kibana 3

  1. Clone the repository from GitHub
    • git clone https://github.com/elasticsearch/kibana.git
  2. Edit the kibana/config.js file and set the elasticsearch line to:
    • elasticsearch:    "http://<your-logstash-public-ip>:9200", 
  3. Copy the Kibana repository to your web server directory
    • CentOS:
      mv kibana/* /var/www/html/; service httpd start
    • Ubuntu:
      mv kibana/* /var/www/

Point your browser to http://<your-logstash-public-ip&gt; and you should be presented with the Kibana interface. Kibana is not specifically a frontend for Logstash but rather a frontend to any ElasticSearch installation. Kibana does provide a default Logstash dashboard as a starting point for you customizations:  http://<your-logstash-public-ip>/index.html#/dashboard/file/logstash.json

Standard
Eucalyptus, QA

Introducing Micro QA

MicroQA-homepage

I have devoted my last 2 years to testing Eucalyptus. In that period the QA team and I have gone through many iterations of tools to find those that make us most efficient. It has become a never ending and enjoyable quest.

We have evolved our testing processes through the following stages:

  1. Using command line tools exclusively
  2. Writing scripts that call command line tools and parsing their output
  3. Writing scripts using a library to make test creation easier and more efficient without the need for command line tools
  4. Running scripts through a graphical tool in order to make test execution more flexible and simple

Each of these iterations was fueled by some tool chain that came along to solve a problem. My journey at Eucalyptus started between the second and third stages. Euca2ools were the go to favorite for manual testing and there was a library aptly named ec2ops floating around that wrapped euca2ools commands using Perl. Being that boto was backing euca2ools at the time I figured I would take a stab at  creating a library calling boto directly that we could start to build our tests with, and thus Eutester was born entering us into the third phase. Once we had reached this point, we were able to quickly write and execute idempotent tests from the command line. After manual execution of our tests was passing consistently we were able to parametrize, run, and parallelize our tests using Jenkins. At this fourth phase we have taken a snapshot of our environment and are now able to share it with the rest of the community through our image catalog.

A few of the use cases Micro QA can help with are:

  • Regression testing during development
  • Functional testing after initial installation
  • Load and stress testing before going into production
  • Development platform for Eutester test cases

Benefits for users of Micro QA include:

  • Known working automated tests
  • Constantly increasing number of test scenarios
  • Flexibility to add custom tests as needed for use cases which aren’t covered

How it works

The environment starts with an Ubuntu Precise Guest Image from Ubuntu’s cloud-images. Once downloaded, registered and started, I installed the Jenkins package. After Jenkins was up and running I installed a redirect for port 8080 to 80 so that users would not need to remember a port in order to access their Micro QA image and could simply hit: http://<instance-public-ip&gt;.  Once the Jenkins instance was reachable by the standard HTTP port (80) we began to add the dependencies for Eutester to the guest OS. The typical environment for Eutester scripts requires boto, paramiko, and virtualenv to isolate the script runtime environment. Once the Python dependencies were successfully installed into a virtualenv we then setup our projects in the Jenkins install. The jobs for Eutester and Eutester4j were then created with only a single required parameter, namely the contents of the eucarc file generated by the cloud. Each script checks out its own environment so that both development and stable Eutester versions can run side by side.

Installation

In order to install the Micro QA image follow the instructions here: https://github.com/eucalyptus/micro-qa/blob/master/README.md

Usage

  1. Once at the main page of the Micro QA instance, click the “Build” button on the right side of the Instance Suite project.
  2. Enter your eucarc file into the text box on the next screen
  3. Hit build at the bottom of the page
  4. You will be taken to the currently running job. Click the blue bar under the job timestamp on the left.
  5. Here you will see the console output for the test run.
  6. When the script completes, the bottom of the console output will display a summary of the results.
Standard
Eucalyptus, QA

Creating a Eucalyptus Test Harness: Jenkins,Testlink and Eutester

Introduction

This post will lead you through the creation of an environment to run automated test plans against a Eucalyptus installation. The tools used were described in my previous post, namely Jenkins (orchestrator), Testlink (planning & metadata), Eutester (functional testing framework). The use case we will be attacking is that of the Developer QA system sandbox.

The only requirements for this setup:

  1. Your machine can run Jenkins
  2. Python 2.6 or greater installed
  3. Python virtualenv is installed (easy_install virtualenv)
  4. Git is installed

Once complete you should be able to run test plans against a running cloud with the click of a button. This has only been tested on my Macbook but should work on any platform that can meet these requirements.

Getting setup

Installing Jenkins

The first step in getting this system up and running is to download and install Jenkins on the desired host machine. When completing this exercise I was using my Mac laptop which led me to use the installer found on the right side of this page: Download Jenkins. Further installation instructions for Linux and Windows can be found here: Installing Jenkins.

Once the installation is complete, point your we browser to port 8080 of the machine you installed Jenkins on, for example http://yourjenkins.machine:8080.  You should now be presented with the main Jenkins dashboard which looks like this:

Installing Jenkins Plugins

The setup we are trying to achieve requires that you install a Jenkins plugin. Luckily for us Jenkins makes this installation a breeze:

  1. Go to your Jenkins URL in a web browser
  2. Click Manage Jenkins on the left
    •  
  3. Click Manage Plugins
  4. Click the Available tab
    •  
  5. Install the following  required plugin by clicking the checkbox on the left of the row on all of them and then clicking “Download now and install after restart” at the bottom of the page
    • Testlink Plugin
  6. Once the installation is complete click the checkbox to restart Jenkins when there are no jobs running (ie now).

Configure Testlink Plugin

In order to access standardized test plans that are implemented by Eutester scripts and cataloged by Testlink we need to setup the Jenkins Testlink plugin as follows:

  1. Click Manage Jenkinson the left
    •  
  2. Click Configure System
  3. Scroll down to the Testlink Section and click the Add button to configure a new Testlink installation to use when pulling test plans and test cases
  4. Configure the options presented as follows:
    • Name: Euca-testlink
    • URL: http://testlink.euca-qa.com/testlink/lib/api/xmlrpc.php
    • Developer Key: 7928ee041d5b20ce3f356992d06ab401
  5. Click Save at the bottom of the page

Now that we have Jenkins and its Testlink plugin configured we need to add a job that uses this functionality. For users not wishing know the internals of the job, instructions follow for importing the pre-built job. For the advanced/curious user the internals of the job are explained in another blog post.

Importing the Pre-configured Job

  1. Go to  your Jenkins home directory (Mac: /Users/Shared/Jenkins/Home, Linux: /var/lib/jenkins)  in a terminal
  2. Create a directory called testlink-testplan in the $JENKINS_HOME/jobs directory
  3. Copy this config file to that directory: Job Config
  4. Change the ownership of the entire testlink-testplan directory to the jenkins user:
    • chown -R jenkins:jenkins $JENKINS_HOME/jobs/testlink-testplan
  5. Go to the Manage Jenkins page then click Reload Configuration from Disk
  6. Upon returning to the main dashboard you should see your job ready to run.
  7. You can now skip down to the section below entitled Running and Viewing Your Test if you do not care to know the internals of the job.

Running and Viewing Your Test

Running the test

  1. In order to run the test job we have just configured return to the Jenkins main dashboard by clicking Jenkinsin the top left corner of the Jenkins page.
  2. Click the Build Now button on the left of the testlink-testplan job 
  3. Ensure that parameters are set properly. The defaults should work other than the topology which will require the IPs of the machines you are trying to test against.

Viewing the test

  1. On the left side of the screen you should see the Build History which should now show at least 1 job running. Look for a build with a blue progress bar under it.

  2. If you would like to see the console output of the currently running test click the blue progress bar.
  3. If you would like to see the main page for the build click the date above the blue progress bar. This screen shows a summary of the test results and any archived artifacts when the build has completed.
  4. The trend button will show a plot of the test duration over time.

You can rinse and repeat as many times as you’d like with this procedure.

Testcases and Testplans

Currently the only testplan available is CloudBasics but soon more testcases and plans will be added. I will make sure to update this blog post with the latest list.

Standard