Introducing Micro QA

I have devoted my last 2 years to testing Eucalyptus. In that period the QA team and I have gone through many iterations of tools to find those that make us most efficient. It has become a never ending and enjoyable quest.

We have evolved our testing processes through the following stages:

1. Using command line tools exclusively
2. Writing scripts that call command line tools and parsing their output
3. Writing scripts using a library to make test creation easier and more efficient without the need for command line tools
4. Running scripts through a graphical tool in order to make test execution more flexible and simple

Each of these iterations was fueled by some tool chain that came along to solve a problem. My journey at Eucalyptus started between the second and third stages. Euca2ools were the go to favorite for manual testing and there was a library aptly named ec2ops floating around that wrapped euca2ools commands using Perl. Being that boto was backing euca2ools at the time I figured I would take a stab at  creating a library calling boto directly that we could start to build our tests with, and thus Eutester was born entering us into the third phase. Once we had reached this point, we were able to quickly write and execute idempotent tests from the command line. After manual execution of our tests was passing consistently we were able to parametrize, run, and parallelize our tests using Jenkins. At this fourth phase we have taken a snapshot of our environment and are now able to share it with the rest of the community through our image catalog.

A few of the use cases Micro QA can help with are:

• Regression testing during development
• Functional testing after initial installation
• Load and stress testing before going into production
• Development platform for Eutester test cases

Benefits for users of Micro QA include:

• Known working automated tests
• Constantly increasing number of test scenarios
• Flexibility to add custom tests as needed for use cases which aren’t covered

How it works

The environment starts with an Ubuntu Precise Guest Image from Ubuntu’s cloud-images. Once downloaded, registered and started, I installed the Jenkins package. After Jenkins was up and running I installed a redirect for port 8080 to 80 so that users would not need to remember a port in order to access their Micro QA image and could simply hit: http://<instance-public-ip&gt;.  Once the Jenkins instance was reachable by the standard HTTP port (80) we began to add the dependencies for Eutester to the guest OS. The typical environment for Eutester scripts requires boto, paramiko, and virtualenv to isolate the script runtime environment. Once the Python dependencies were successfully installed into a virtualenv we then setup our projects in the Jenkins install. The jobs for Eutester and Eutester4j were then created with only a single required parameter, namely the contents of the eucarc file generated by the cloud. Each script checks out its own environment so that both development and stable Eutester versions can run side by side.

Installation

In order to install the Micro QA image:

1. Download image from emis-catalog.eucalyptus-systems.com:  Direct Link
2. Upload the image to your cloud using the following command from eustore:
   • eustore-install-image -t micro-qa-v2.tgz -a x86_64 -s “Micro QA v2″ -b micro-qa-v2 -k kvm
3. After the image is uploaded and registered, run an instance of the image with at least 1GB of RAM
4. Login to the instance by pointing a browser to: http://<instance-public-ip&gt;

Usage

1. Once at the main page of the Micro QA instance, click the “Build” button on the right side of the Instance Suite project.
2. Enter your eucarc file into the text box on the next screen
3. Hit build at the bottom of the page
4. You will be taken to the currently running job. Click the blue bar under the job timestamp on the left.
5. Here you will see the console output for the test run.
6. When the script completes, the bottom of the console output will display a summary of the results.

Deploying Eucalyptus via Ansible playbook(s)

Reblogged from Take that to the bank and cash it!:

The first cut of the Ansible deployment playbook for deploying Eucalyptus private clouds is ready.  I've merged the first "release" into the master branch here: https://github.com/lwade/eucalyptus-playbook. Feedback and contributions are very welcome, please file issues against the project.

This playbook allows a user to deploy a single front-end cloud (i.e. all component on a single system) and as many NC's as they want. 

Read more… 694 more words

Nice work by Lester! This will make deployment of your Euca cloud much much easier!

Test Drive: Drupal Deployment on Eucalyptus using Stackato, Amazon Route 53 and the Eucalyptus Community Cloud

Reblogged from More Mind Spew-age from Harold Spencer Jr.:

• 

Recently, I did a blog discussing how to deploy a Jenkins server using Stackato, running on Eucalyptus. At the end of that blog, I mentioned how the Eucalyptus Community Cloud (ECC) could be used for testing out the Stackato Microcloud image on Eucalyptus. The previous blog - I felt - was more for DevOps administrators who had access to their own on-premise Eucalyptus clouds.

Read more… 790 more words

Beautiful use case for Eucalyptus.

Introducing Metaleuca

Reblogged from Kyo Lee:

• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 
• 

What is Metaleuca?

Metaleuca is a bare-metal provision management system that interacts with open-source software Cobbler via EC2-like CLI.

Using Metaleuca, users can communicate with Cobbler to self-provision a group of bare-metal machines to boot up with new, fresh OS images. The main appeal of Metaleuca is that it allows users to manage the bare-metal machines like EC2's virtual instances via the command-lines that feel much like ec2-tools, or euca2ools.

Read more… 838 more words

Very nice implementation of cobbler CLI tools. Great work Kyo!!

Eucalyptus and Nagios

Reblogged from nurmiblog:

Production deployments of Eucalyptus, like production deployments of any infrastructure software running in a data center, require some amount of health and status monitoring be happening in order to both allow the Eucalyptus/data-center administrator the ability to stay on top of evolving resource situations and to provide invaluable diagnostic information when something is going sideways within the resource pool.  Fortunately for all of us, there exists a wide variety of health/status monitoring system out there, and several of them are of extremely high quality, tried and tested, and are available as part of major Linux distributions as pre-packaged open-source solutions.  

Read more… 1,429 more words

This will certainly become a tool that I use day to day. Thanks nurmi!

Using Scalr for Automation of your Eucalyptus Cloud

Introduction

 

I have been using Eucalyptus heavily (as a quality engineer it is my day to day) for the past 1.5 years. I know the ins and the outs of system and am constantly tracking new features and bug fixes that arrive. With this knowledge it makes me a prime candidate to find out how other pieces of the cloud story can integrate with Eucalyptus.

I run a small cloud at home that I use for development and testing of different software stacks. Some of the tools that I’ve learned to use and hack on since Ive turned up my cloud include Graphite, Gitorious, Jenkins, Testlink, Zenoss. The issue with getting most of these (and any) open-source tools running is that they often require a very particular  base OS and dependency versions in order to install cleanly. This makes Eucalyptus a great tool for figuring out the right/easiest way to deploy an open source tool as it allows users to create and save any particular stack they have created for later use. Another great part about using Eucalyptus as a development tool is that it allows any and all distros to be loaded into a cloud and available for use by many users. When I look into a new tool to use I can rest assured that I can find one of their supported distros in my Eucalyptus cloud. Currently I have images registered for Debian 5/6, Ubuntu 12.04/12.10/13.04, Fedora 17, CentOS 6, and begrudgingly Windows 7 (99% used to manage vCenter nodes yuck).

There are many ways to populate and provision application stacks in a cloudy model. One way would be to load all of the application stack onto an image then re-bundle it and save it off. Another approach, and the one that I rely on, is to populate base images that can then be provisioned by scripts in order to run the application of choice. In using this model I had come up with many different scripts/user-data to populate the images listed above with applications. My approach allowed for apps to be deployed easily but I ended up with lots of replicated code across the user-data that I was passing to instances. Another pain was that it required me to remember which user-data scripts belonged with which images. Although not impossible to turn up and down my apps, it was definitely cumbersome and certainly not push button.

Enter Scalr.

Scalr allows me to populate scripts into a common database that is shared across all images/clouds. Because Scalr allows multiple scripts to be run during instance turn up (and manually for that matter), I am now able to modularize my scripts to reduce the amount of code I need to maintain.  Another element of Scalr that makes my life easier is its ability to auto scale an application. For my purposes I am not scaling an app to more than 1 instance, but if I ever bring down an instance erroneously Scalr will automatically repopulate the instance and run the scripts necessary to reconstruct the app. The final efficiency I get out of using Scalr is that I can load up multiple clouds and use the same scripts against similar base images within various clouds. As a tester I bring up and down at least 1 private cloud a day, on top of using my AWS account for both testing and production use, so this may be the last feature but certainly not least in my mind.

Eucalyptus+Scalr is a developer/tester’s dream, so lets get down to business on how to get this solution running on your own cloud.

Install Scalr

1. Run the script found here on a Ubuntu Precise 12.04 server or instance: https://gist.github.com/4527791
2. You will be prompted a few times to enter the password listed in the script (default is yourpassword)
3. Ensure that the default virtual host (scalr.local) can be resolved by either adding a DNS entry to the server or a host entry on your client. Default hostname can be changed on lines 4-5 in the script from step 1.
4. Login to Scalr as admin/admin: http://scalr.local
5. Click on Accounts in the top left corner and create a user account
6. Click Settings->Core Settings, and set the “Event handler URL” at the bottom to the hostname you chose in step 3 (default is sclar.local)
7. Logout and back in with your user credentials created in step 5

Scalrize Your Images

You need to install an agent, Scalarizr, on any  any images used to create server templates, or in Scalr parlance roles. After installing the Scalarizr agent you will register each of your images as a role. To install Scalarizr:

1. Install the repository package as appropriate for your distro: Debian  RHEL/CentOS
2. Install the Scalarizr package appropriate to the cloud that will be used. In our case “scalarizr-eucalyptus”
3. Ensure that your image can properly resolve the “Event Handler URL” entered when installing Scalr
4. Rebundle and register the image with Eucalyptus

Register Your Eucalyptus Cloud

Scalr works with many cloud providers including AWS. They were able to leverage a good amount of  client code  in order to support Eucalyptus. It would seem that the last time that the Eucalyptus integration was looked at was in the Euca 2.x timeframe so many things that Eucalyptus now has full support for with 3.1+ (EBS backed images for example) are not supported. Other missing functionality is support for keypairs (which can be patched using scripts in Scalr) and all instances are launched in the default group (not sure what the reason for this is).

In order to setup Scalr with your cloud credentials:

1. In the top left of the navigation bar click “default”, the “Manage”
2. Click “Actions” on the right, then “Configure”
3. Set your timezone
4. Click the Eucalyptus logo
5. Click the green plus sign to add a new Eucalyptus cloud

Creating a Role

Scalr uses a paradigm for cloud automation that requires that cloud images be registered as Roles. These roles are then added to Farms in order to deploy an app. Each role can only appear once per Farm. Scalr allows you to catalog, version and deploy scripts using a templating mechanism where values can be set at the Role, Farm, or individual server level.

Some common role types are:

1. Base Images
2. Load Balancers
3. Web Servers

In order to create a Role:

1. Go to this page, replacing the hostname if necessary: http://scalr.local/#/roles/edit
2. Once there, enter a name for the role, for example: Ubuntu Precise Base
3. Click on the check box that represents what category of role this is, in our case we will check the checkbox for Base
4. Click over to the Images tab and enter the information about the image you registered in the steps above
5. Enter the image info including its machine image ID
6. Click Add (left side) then Save (bottom center)

Adding your scripts

One of the most powerful parts about Scalr is the ability to write and reuse templated scripts. Scalr also allows you to share, fork, and version your scripts. Scripts can be added to Roles, Farms or individual servers for execution at boot, termination or manually during any part of an instances’ lifecycle. Creating, managing, and deploying scripts allows you to work around some shortcomings of the current Scalr/Eucalyptus integration like only being able to use the default security-group and no keypair being passed to instances launched through Scalr.

1. Go to http://scalr.local/#/scripts/view in order to add some basic scripts 
2. Click the green plus sign on the right of the Scalr Web Console
3. When adding scripts you will need to give them a name and a description along with the actual code
4. The first script we will add will be called “SSH Key Inject” and ensure that our SSH key can be added to an instance: 

   #!/bin/bash
   echo %ssh_key% > /root/.ssh/authorized_keys
   

5. The next script we will add will install Graphite, a scalable realtime graphing application and can be found here

You will notice that the script added in step 2 uses a wild card parameter, %ssh_key%, that can be configured at script run time, role provisioning or farm launch time. Once we’ve created our scripts its time to create our farm which will pair our scripts with our images, or Roles, so that we can run and terminate our application at will.

Creating a Farm

Farms are collections of roles that constitute a single deployment. All servers in a farm can be turned up and down in unison in order to deploy an application. Individual roles within a farm can be autoscaled. When Scalr notices less than a certain threshold of servers in a particular role, it will automatically launch more servers. As an example Farm, I will be showing how to deploy Graphite (Scalable Realtime Graphing).

1. Add a role using the procedure above that references a Scalarized Precise Base Image 
2. Click over to the Farms pane in the Scalr web interface
3. Click the green plus sign on the right of the Scalr Web Console
4. Enter a name for this farm, in our case: Graphite
5. Click over to the Roles tab and then click on Roles Library
6. Click the plus sign next to Base Images
7. Click on the icon for the Precise Base Image, then click the green plus sign.

Now that we have added the Base Image role to the Farm we will need to add scripts to the Role that run once the instance is up and running.

1. Click on the icon that has now appeared towards the top of the page in order to configure what scripts we will run on this Role in the Farm
2. On the bottom left click Scripting. In this configuration page we can choose which scripts to run, in which order and at what times
3. Under the “When” dropdown click “Host Up”, then under the “execute” drop down choose the “SSH Key Inject” script we added in the steps from above. Once those two options have been chosen, click the green plus sign
4.   Click on the row that has shown up for that script
5. On the right side of the pane:
   1. Ensure that “Where” is set to “All instances of this role”
   2. Under the parameters section at the bottom of the pane, set which public key we want to inject for this role
6. Add our second script “Install Graphite” at “Host Up”, ensure that “Where” is set to “All instances of this role”
7. Click save at the bottom of the interface

Once we have defined what roles and scripts are paired up to make our application we can launch our Farm.

1. Go to the Farms interface: http://scalr.local/#/farms
2. Click Actions on the right side page
3. Click Launch

Scalr will now launch your image and run the desired scripts to build out the application. You can use this interface to turn up and down your applications as necessary. If Scalr notices that your app is no longer running on the cloud it will be automatically relaunched for you. I have used this feature many times to help me with disaster recovery. Once I have an application running properly in my cloud I ensure that I can terminate any individual instances and my Scalr configuration is setup to properly rebuild the app.

Configuring the Eucalyptus User Console with a Reverse Proxy

Reblogged from Coders Like Us:

• 

The Eucalyptus User Console can be used standalone, but generally people run Tornado apps (as this is) behind a reverse proxy. There are a few reasons, but most commonly, it is so SSL termination can be handled in one place and several Tornado instances can be managed behind one front end. FriendFeed (who developed Tornado) talked about configuring one Tornado instance per core behind Nginx as the reverse proxy.

Read more… 991 more words

Badassness