EucaLoader: Load Testing Your Eucalyptus Cloud

Introduction

After provisioning a cloud that will be used by many users, it is best practice to do load or burn in testing to ensure that it meets your stability and scale requirements. These activities can be performed manually by running commands to run many  instances or create many volumes for example. In order to perform sustained long term tests it is beneficial to have an automated tool that will not only perform the test actions but also allow you to analyze and interpret the results in a simple way.

---

Background

Over the last year, I have been working with Locust to provide a load testing framework for Eucalyptus clouds. Locust is generally used for load testing web pages but allows for customizable clients which allowed me to hook in our Eutester library in order to generate load. Once I had created my client, I was able to create Locust “tasks” that map to activities on the cloud. Tasks are user interactions like creating a bucket or deleting a volume. Once the tasks were defined I was able to compose them into user profiles that define which types of actions each simulated user will be able to run as well as weighting their probability so that the load can most closely approximate a real world use case. In order to make the deployment of EucaLoader as simple as possible, I have baked the entire deployment into a CloudFormation template. This means that once you have the basics of your deployment done, you can start stressing your cloud and analyzing the results with minimal effort.

---

Using EucaLoader

Prerequisites

In order to use EucaLoader you will first need to load up an Ubuntu Trusty image into your cloud as follows:

# wget https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
# euca-install-image -i trusty-server-cloudimg-amd64-disk1.raw -n trusty -r x86_64 -b trusty --virt hvm

We will also need to clone the EucaLoader repository and install its dependencies:

# git clone https://github.com/viglesiasce/euca-loader
# pip install troposphere

Next we will upload credentials for a test account to our objectstore so that our loader can pull them down for Eutester to use:

# euare-accountcreate loader
# euca_conf --get-credentials  loader.zip --cred-account loader
# s3cmd mb s3://loader
# s3cmd put -P loader.zip s3://loader/admin.zip

---

Launching the stack

Once inside the euca-loader directory we will create our CloudFormation template and then create our stack by passing in the required parameters:

# ./create-locust-cfn-template.py > loader.cfn
# euform-create-stack --template-f loader.cfn  loader -p KeyName=<your-keypair-name> -p CredentialURL='http://<your-user-facing-service-ip>:8773/services/objectstorage/loader/admin.zip' -p ImageID=<emi-id-for-trusty> -p InstanceType=m1.large

At this point you should be able to monitor the stack creation with the following commands

# euform-describe-stacks
# euform-describe-stack-events loader

Once the stack shows as CREATE_COMPLETE, the describe stacks command should show outputs which point you to the Locust web portal (WebPortalUrl) and to your Grafana dashboard for monitoring trends (GrafanaURL).

---

Starting the tests

In order to start your user simulation, point your web browser to the the WebPortalUrl as defined by the describe stacks output. Once there you can enter the amount of users you’d like to simulate as well as how quickly those users should “hatch”.

Once you’ve started the test, the statistics for each type of requests will begin to show up in the Locust dashboard.

---

See your results

In order to better visualize the trends in your results, EucaLoader provides a Grafana dashboard that tracks a few of the requests for various metrics. This dashboard is easily customized to your particular test and is meant as a jumping off point.

Creating a Eucalyptus Test Harness: Jenkins,Testlink and Eutester

Introduction

This post will lead you through the creation of an environment to run automated test plans against a Eucalyptus installation. The tools used were described in my previous post, namely Jenkins (orchestrator), Testlink (planning & metadata), Eutester (functional testing framework). The use case we will be attacking is that of the Developer QA system sandbox.

The only requirements for this setup:

1. Your machine can run Jenkins
2. Python 2.6 or greater installed
3. Python virtualenv is installed (easy_install virtualenv)
4. Git is installed

Once complete you should be able to run test plans against a running cloud with the click of a button. This has only been tested on my Macbook but should work on any platform that can meet these requirements.

Getting setup

Installing Jenkins

The first step in getting this system up and running is to download and install Jenkins on the desired host machine. When completing this exercise I was using my Mac laptop which led me to use the installer found on the right side of this page: Download Jenkins. Further installation instructions for Linux and Windows can be found here: Installing Jenkins.

Once the installation is complete, point your we browser to port 8080 of the machine you installed Jenkins on, for example http://yourjenkins.machine:8080.  You should now be presented with the main Jenkins dashboard which looks like this:

Installing Jenkins Plugins

The setup we are trying to achieve requires that you install a Jenkins plugin. Luckily for us Jenkins makes this installation a breeze:

1. Go to your Jenkins URL in a web browser
2. Click Manage Jenkins on the left
   •  
3. Click Manage Plugins
   • 
4. Click the Available tab
   •  
5. Install the following  required plugin by clicking the checkbox on the left of the row on all of them and then clicking “Download now and install after restart” at the bottom of the page
   • Testlink Plugin
   • 
6. Once the installation is complete click the checkbox to restart Jenkins when there are no jobs running (ie now).

Configure Testlink Plugin

In order to access standardized test plans that are implemented by Eutester scripts and cataloged by Testlink we need to setup the Jenkins Testlink plugin as follows:

1. Click Manage Jenkinson the left
   •  
2. Click Configure System
   • 
3. Scroll down to the Testlink Section and click the Add button to configure a new Testlink installation to use when pulling test plans and test cases
4. Configure the options presented as follows:
   • Name: Euca-testlink
   • URL: http://testlink.euca-qa.com/testlink/lib/api/xmlrpc.php
   • Developer Key: 7928ee041d5b20ce3f356992d06ab401
   • 
5. Click Save at the bottom of the page

Now that we have Jenkins and its Testlink plugin configured we need to add a job that uses this functionality. For users not wishing know the internals of the job, instructions follow for importing the pre-built job. For the advanced/curious user the internals of the job are explained in another blog post.

Importing the Pre-configured Job

1. Go to  your Jenkins home directory (Mac: /Users/Shared/Jenkins/Home, Linux: /var/lib/jenkins)  in a terminal
2. Create a directory called testlink-testplan in the $JENKINS_HOME/jobs directory
3. Copy this config file to that directory: Job Config
4. Change the ownership of the entire testlink-testplan directory to the jenkins user:
   • chown -R jenkins:jenkins $JENKINS_HOME/jobs/testlink-testplan
5. Go to the Manage Jenkins page then click Reload Configuration from Disk
6. Upon returning to the main dashboard you should see your job ready to run.
7. You can now skip down to the section below entitled Running and Viewing Your Test if you do not care to know the internals of the job.

Running and Viewing Your Test

Running the test

1. In order to run the test job we have just configured return to the Jenkins main dashboard by clicking Jenkinsin the top left corner of the Jenkins page.
   • 
2. Click the Build Now button on the left of the testlink-testplan job 
3. Ensure that parameters are set properly. The defaults should work other than the topology which will require the IPs of the machines you are trying to test against.

Viewing the test

1. On the left side of the screen you should see the Build History which should now show at least 1 job running. Look for a build with a blue progress bar under it.
   
   • 
2. If you would like to see the console output of the currently running test click the blue progress bar.
3. If you would like to see the main page for the build click the date above the blue progress bar. This screen shows a summary of the test results and any archived artifacts when the build has completed.
   • 
4. The trend button will show a plot of the test duration over time.

You can rinse and repeat as many times as you’d like with this procedure.

Testcases and Testplans

Currently the only testplan available is CloudBasics but soon more testcases and plans will be added. I will make sure to update this blog post with the latest list.