Eucalyptus, QA

EucaLoader: Load Testing Your Eucalyptus Cloud

Locust-full-page

Introduction

After provisioning a cloud that will be used by many users, it is best practice to do load or burn in testing to ensure that it meets your stability and scale requirements. These activities can be performed manually by running commands to run many  instances or create many volumes for example. In order to perform sustained long term tests it is beneficial to have an automated tool that will not only perform the test actions but also allow you to analyze and interpret the results in a simple way.

Background

Over the last year, I have been working with Locust to provide a load testing framework for Eucalyptus clouds. Locust is generally used for load testing web pages but allows for customizable clients which allowed me to hook in our Eutester library in order to generate load. Once I had created my client, I was able to create Locust “tasks” that map to activities on the cloud. Tasks are user interactions like creating a bucket or deleting a volume. Once the tasks were defined I was able to compose them into user profiles that define which types of actions each simulated user will be able to run as well as weighting their probability so that the load can most closely approximate a real world use case. In order to make the deployment of EucaLoader as simple as possible, I have baked the entire deployment into a CloudFormation template. This means that once you have the basics of your deployment done, you can start stressing your cloud and analyzing the results with minimal effort.

Using EucaLoader

Prerequisites

In order to use EucaLoader you will first need to load up an Ubuntu Trusty image into your cloud as follows:

# wget https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
# euca-install-image -i trusty-server-cloudimg-amd64-disk1.raw -n trusty -r x86_64 -b trusty --virt hvm

We will also need to clone the EucaLoader repository and install its dependencies:

# git clone https://github.com/viglesiasce/euca-loader
# pip install troposphere

Next we will upload credentials for a test account to our objectstore so that our loader can pull them down for Eutester to use:

# euare-accountcreate loader
# euca_conf --get-credentials  loader.zip --cred-account loader
# s3cmd mb s3://loader
# s3cmd put -P loader.zip s3://loader/admin.zip


Launching the stack

Once inside the euca-loader directory we will create our CloudFormation template and then create our stack by passing in the required parameters:

# ./create-locust-cfn-template.py > loader.cfn
# euform-create-stack --template-f loader.cfn  loader -p KeyName=<your-keypair-name> -p CredentialURL='http://<your-user-facing-service-ip>:8773/services/objectstorage/loader/admin.zip' -p ImageID=<emi-id-for-trusty> -p InstanceType=m1.large

At this point you should be able to monitor the stack creation with the following commands

# euform-describe-stacks
# euform-describe-stack-events loader

Once the stack shows as CREATE_COMPLETE, the describe stacks command should show outputs which point you to the Locust web portal (WebPortalUrl) and to your Grafana dashboard for monitoring trends (GrafanaURL).


Starting the tests

In order to start your user simulation, point your web browser to the the WebPortalUrl as defined by the describe stacks output. Once there you can enter the amount of users you’d like to simulate as well as how quickly those users should “hatch”.

Locust-start-test

Once you’ve started the test, the statistics for each type of requests will begin to show up in the Locust dashboard.

Locust-test-running


See your results

In order to better visualize the trends in your results, EucaLoader provides a Grafana dashboard that tracks a few of the requests for various metrics. This dashboard is easily customized to your particular test and is meant as a jumping off point.

Locust-dashboard

Standard
Eucalyptus

Deploying Cassandra and Consul with Chef Provisioning

ConsulCassandra

Introduction

Chef Provisioning (née Chef Metal) is an incredibly flexible way to deploy infrastructure. Its many plugins allow users to develop a single methodology for deploying an application that can then be repeated against many types of infrastructure (AWS, Euca, Openstack, etc). Chef provisioning is especially useful when deploying clusters of machines that make up an application as it allows for machines to be:

  • Staged before deployment
  • Batched for parallelism
  • Deployed in serial when necessary

This level of flexibility means that deploying interesting distributed systems like Cassandra and Consul is a breeze. By leveraging community cookbooks for Consul and Cassandra, we can largely ignore the details of package installation and service management and focus our time on orchestrating the stack in the correct order and configuring the necessary attributes such that our cluster converges properly. For this tutorial we will be deploying:

  • DataStax Cassandra 2.0.x
  • Consul
    • Service discovery via DNS
    • Health checks on a per node basis
  • Consul UI
    • Allows for service health visualization

Once complete we will be able to use Consul’s DNS service to load balance our Cassandra client requests across the cluster as well as use Consul UI in order to keep tabs on our clusters’ health.

In the process of writing up this methodology, I went a step further and created a repository and toolchain for configuring and managing the lifecycle of clustered deployments. The chef-provisioning-recipes repository will allow you to configure your AWS/Euca cloud credentials and images and deploy any of the clustered applications available in the repository.

Steps to reproduce

Install prerequisites

  • Install ChefDK
  • Install package deps (for CentOS 6)
    yum install python-devel gcc git
  • Install python deps:
    easy_install fabric PyYaml
  • Clone the chef-provisioning-recipes repo:
    git clone https://github.com/viglesiasce/chef-provisioning-recipes

Edit config file

The configuration file (config.yml) contains information about how and where to deploy the cluster. There are two main sections in the file:

  1. Profiles
    1. Which credentials/cloud to use
    2. What image to use
    3. What instance type to use
    4. What username to use
  2. Credentials
    1. Cloud endpoints or region
    2. Cloud access and secret keys

Edit the config.yml file found in the repo such that the default profile points to a CentOS 6 image in your cloud and the default credentials point to the proper cloud.

Run the deployment

Once the deployer has been configured we simply need to run it and tell it which cluster we would like to deploy. In this case we’d like to deploy Cassandra so we will run the deployer as follows:

./deployer.py cassandra

This will now automate the following process:

  1. Create a chef repository
  2. Download all necessary cookbooks
  3. Create all necessary instances
  4. Deploy Cassandra and Consul

Once this is complete you should be able to see your instances running in your cloud tagged as follows: cassandra-default-N. In order to access your Consul UI dashboard go to http://instance-pub-ip:8500

You should now also be able to query any of your Consul servers for the IPs of your Cassandra cluster:

nslookup cassandra.service.paas.home &amp;amp;amp;lt;instance-pub-ip&amp;amp;amp;gt;

In order to tear down the cluster simply run:

./deployer.py cassandra --op destroy
Standard
Eucalyptus

Chef Metal with Eucalyptus

Introduction

My pull request to chef-metal-fog was recently accepted and released in version 0.8.0 so a quick post on how to get up and running on your Eucalyptus Cloud seemed appropriate.

Chef Metal is a new way to provision your infrastructure using Chef recipes. It allows you to use the same convergent design as normal Chef recipes. You can now define your cloud or bare metal deployment in a Chef recipe then deploy, update and destroy it with chef-client. This flexibility is incredibly useful for both development of new Chef cookbooks and in exploring various topologies of distributed systems.

Game time

First, install the Chef Development Kit. This will install chef-client and a few other tools to get you well on your way to Chef bliss.

Once you have installed the Chef DK on your workstation, install the chef-metal gem into the Chef Ruby environment:

chef gem install chef-metal

You will need to create your Chef repo. This repository will contain all the information about how and where your application gets deployed using Chef Metal. In this case we are naming our app “euca-metal”.

chef generate app euca-metal

You should now see a directory structure as follows:

├── README.md
└── cookbooks
 └── euca-metal
   ├── Berksfile
   ├── chefignore
   ├── metadata.rb
   └── recipes
     └── default.rb

Now that the skeleton of our application has been created lets edit cookbooks/euca-metal/recipes/default.rb to look like this:

require 'chef_metal_fog'

### Arbitrary name of our deployment
deployment_name ='chef-metal-test'

### Use the AWS provider to provision the machines
### Here is where we set our endpoint URLs and keys for our Eucalyptus deployment
with_driver 'fog:AWS', :compute_options => { :aws_access_key_id => 'XXXXXXXXXXXXXXX',
                                             :aws_secret_access_key => 'YYYYYYYYYYYYYYYYYYYYYYYYYY',
                                             :ec2_endpoint => 'http://compute.cloud:8773/services/compute',
                                             :iam_endpoint => 'http://euare.cloud:8773/services/objectstorage'
}

### Create a keypair named after our deployment
fog_key_pair deployment_name do
  allow_overwrite true
end

### Use the key created above to login as root, all machines below
### will be run using these options
with_machine_options ssh_username: 'root', ssh_timeout: 60, :bootstrap_options => {
  :image_id => 'emi-A6EA57D5',
  :flavor_id => 't1.micro',
  :key_name => deployment_name
}

### Launch an instance and name it after our deployment
machine deployment_name do
  ### Install Java on the instance using the Java recipe
  recipe 'java'
end

Once we have defined our deployment we will need to create a local configuration file for chef-client:

mkdir -p .chef; echo 'local_mode true' > .chef/knife.rb

Now that we have defined the deployment and setup chef-client, lets run the damn thing!

chef-client -z cookbooks/euca-metal/recipes/default.rb

You can now see Chef create your keypair, launch your instance, and then attempt to run the “java” recipe as we specified. Unfortunately this has failed. We never told our euca-metal cookbook that it required the Java cookbook nor did we download that cookbook for it to use. Let’s fix that.

First we will tell our euca-metal cookbook that we need it to pull in the ‘java’ cookbook in order to provision the node. We need to add the ‘depends’ line to our cookbook’s metadata.rb file which can be found here: cookbooks/euca-metal/metadata.rb

name 'euca-metal'
maintainer ''
maintainer_email ''
license ''
description 'Installs/Configures euca-metal'
long_description 'Installs/Configures euca-metal'
version '0.1.0'
depends 'java'

Next we will need to actually download that Java cookbook that we now depend on. To do that we need to:

# Change to the euca-metal cookbook directory
cd cookbooks/euca-metal/
# Use berkshelf to download our cookbook dependencies
berks vendor
# Move the berks downloaded cookbooks to our main cookbook repository
# Note that it wont overwrite our euca-metal cookbook
mv berks-cookbooks/* ..
cd ../..
# Rerun our chef-client to deploy Java for realz
chef-client -z cookbooks/euca-metal/recipes/default.rb

You will notice that the machine is not reprovisioned (YAY convergence!). The Java recipe should now be running happily on your existing instance. You can find your ssh keys in the .chef/keys directory.

Happy AWS Compatible Private Cloud Cheffing!!!!

Many thanks to John Keiser for his great work on chef-metal.

Standard
Eucalyptus

Using Comcast CMB for SQS and SNS on Eucalyptus

Introduction

As part of a service oriented infrastructure there comes a need to coordinate work between services. AWS provides a couple of services which allow for application components to communicate with each other and their users/administrators in a decoupled fashion.

The Simple Queue Service (SQS) is a mechanism for an applications producers to distribute work to their consumers in a scalable, reliable and fault tolerant way. The basic lifecycle in SQS is as follows:

  1. A queue is created
  2. Producers send arbitrary text messages into the queue
  3. Consumers are constantly listing the messages in a queue and when one is available they “check out” the work by reading the message
  4. Once the message is read a timer kicks that makes the message unreadable by other consumers for a certain period of time (called the visibility timeout).
  5. The consumer can then perform the necessary task described by the message and then delete the message from the queue
  6. If the consumer does not complete the task in time or fails for some other reason the message is made visible again in the queue and picked up by another consumer

One simple example of using this service would be for a Web application front end to take image conversion orders from a user and then throw the image conversion task into a queue that can then be serviced by a fleet of worker nodes that do the actual image processing (ie the compute heavy portion).

The Simple Notification Service (SNS) is a service that allows for the coordination of messages that have one or more recipient subscribing endpoints. In this service users create a topic and then other services and users can subscribe to the topic in order to receive notifications about its goings on. In this model the sender of the message does not have to know where messages are actually being sent but rather that all subscribers (ie people/apps who need the message) will receive the message in the form that they have requested. Subscriptions to topics can be made through various transport mechanisms:

  1. HTTP
  2. HTTPS
  3. SMS
  4. Email
  5. Email-json
  6. SQS

By publishing a message to a topic with multiple subscribers you can ensure that both applications and the people managing them are all on the same page.

Eucalyptus currently does not implement SQS and SNS but the folks over at Comcast have created an incredibly useful open source project that mirrors the APIs with absolutely incredible fidelity. Not only did they ensure that their API coverage was accurate and useful but they built the application stack on top of Cassandra and Redis making it not only horizontally scalable but extremely performant to boot. For more information: Comcast CMB.

Running CMB in your Eucalyptus cloud

In order to simplify the process of installing and bootstrapping CMB, I have created an image that you can install on your cloud with all the requisite services in place. All instructions here should be performed from your Eucalyptus CLC with your admin credentials sourced.

  1. Download the image and decompress it
    1. curl http://eucalyptus-images.s3.amazonaws.com/public/cmb.raw.xz > cmb.raw.xz
    2. xz -d cmb.raw.xz
  2. Install the image
    1. euca-install-image –virt hvm -i cmb.raw -r x86_64 -b CMB -n CMB
  3. Launch the image
    1. euca-run-instance -k <my-keypair> <emi-from-step-2>
  4. Once the image is launched login to the admin portal to create your first user and get your credentials
    1. Goto http://<instance-public-ip&gt;:6059/webui
    2. Login with: cns_internal/cns_internal
    3. Create a new user
    4. Take note of the Access and Secret keys for your new users
  5. Start using your new services with your favorite SDK

Example: Interacting with SQS using Boto

In the example below swap change the following variables to fit your environment:

  • cmb_host – Hostname or IP of your CMB server
  • access_key – Taken from step 4D above
  • secret_key – Taken from step 4D above
#!/usr/bin/python
from boto.sqs.regioninfo import SQSRegionInfo
from boto.sqs.connection import SQSConnection

cmb_host = 'instance-ip'
access_key = 'your-access-key-from-step-4D'
secret_key = 'your-secret-key-from-step-4D'
cmb_sqs_port = 6059

sqs_region = SQSRegionInfo(endpoint=cmb_host, name='home')
cmb_sqs = SQSConnection(aws_access_key_id=access_key, aws_secret_access_key=secret_key,
region=sqs_region, is_secure=False,
port=cmb_sqs_port)

queue = cmb_sqs.create_queue('test')
msg = queue.new_message('Hello World')
queue.write(msg)

all_queues = cmb_sqs.get_all_queues()
print 'Current queues: '  + str(all_queues)
for queue in all_queues:
    print 'Messages in queue: ' + str([msg.get_body() for msg in queue.get_messages()])
Standard
Eucalyptus

Install Eucalyptus 4.0 Using Motherbrain and Chef

 

Introduction

Installing distributed systems can be a tedious and time consuming process. Luckily there are many solutions for distributed configuration management available to the open source community. Over the past few months, I have been working on the Eucalyptus cookbook which allows for standardized deployments of Eucalyptus using Chef. This functionality has already been implemented in MicroQA using individual calls to Knife (the Chef command line interface) for each machine in the deployment. Orchestration of the deployment is rather static and thus only 3 topologies have been implemented as part of the deployment tab.

Last month, Riot Games released Motherbrain, their orchestration framework that allows flexible, repeatable, and scalable deployment of multi-tiered applications. Their approach to the deployment roll out problem is simple and understandable. You configure manifests that define how your application components are split up then define the order in which they should be deployed.

For example in the case of Eucalyptus we have cluster, node, and frontend components. Each component is a set of recipes from the Eucalyptus cookbook. Once we have recipes mapped to components we need to define the order in which these components should be rolled out in the “stack order” section of our Motherbrain manifest:

stack_order do
bootstrap ‘cloud::full’
bootstrap ‘cloud::default’
bootstrap ‘cloud::frontend’
bootstrap ‘cluster::default’
bootstrap ‘cluster::cluster-controller’
bootstrap ‘cluster::storage-controller’
bootstrap ‘cloud::user-facing’
bootstrap ‘cloud::walrus’
bootstrap ‘cloud::user-console’
bootstrap ‘node::default’
bootstrap ‘cloud::configure’
bootstrap ‘nuke::default’
end

Once we have the components split up and ordered we need to define our topology. This can we done with another JSON formatted manifest like so:

{“nodes”: [
{ “groups”: [“cloud::frontend”, “cloud::configure”],
“hosts”: [“10.0.1.185”]
},
{ “groups”: [“cluster::default”],
“hosts”: [“10.0.1.186”]
},
{ “groups”: [“node::default”],
“hosts”: [“10.0.1.187”, “10.0.1.181”]
}]

}

With this information, Motherbrain allows you to create arbitrary topologies of your distributed system with repeatability and scalability taken care of. Repeatability comes from using Chef recipes and the scalability is derived from the nodes in each tier being deployed in parallel. In Eucalyptus terms, this means that no matter how many Node Controllers you’d like to deploy to your cluster, they system will come up in almost constant time. In order to tweak the configuration you can deploy your stack into a properly parameterized Chef environment.
Now that the concept has been laid out, lets get to business building our cluster from the 4.0 nightlies.

Installing prerequisites

I have created a script to install and configure Motherbrain and Chef that should work for Enterprise Linux or Mac OSX:

sh <(curl -s https://gist.githubusercontent.com/viglesiasce/9734682/raw/install_motherbrain.sh)

If you’d like to do the steps manually you can:

  1. Install ruby 2.0.0
  2. Install gems
    1. chef
    2. motherbrain
    3. chef-zero
  3. Get cookbooks and dependencies
    1. eucalyptus – https://github.com/eucalyptus/eucalyptus-cookbook
    2. ntp – https://github.com/opscode-cookbooks/ntp.git
    3. selinux – https://github.com/opscode-cookbooks/selinux.git
    4. yum – https://github.com/opscode-cookbooks/yum.git
  4. Upload all cookbooks to your Chef server
  5. Configure Motherbrain
    1. mb configure

Customizing your deployment

  1. Go into the Eucalyptus cookbook directory (~/chef-repo/cookbooks/eucalyptus)
  2. Edit the bootstrap.json file to match your deployment topology
    1. Ensure at least 1 IP/Machine for each component
    2. Same IP can be used for all machines (Cloud-in-a-box)
  3. Edit the environment file in ~/chef-repo/cookbooks/eucalyptus/environments/edge-nightly.json
    1. Change the topology configuration to match what you have defined in the bootstrap.json file
    2. Change the network config to match your Eucalyptus deployment
  4. Upload your environment to the Chef server
    1. knife environment from file environments/edge-nightly.json

Deploying your Eucalyptus Cloud

Now that we have defined our topology and network configuration we can deploy the cookbook using the Motherbrain command line interface by telling the tool:

  1. Which bootstrap configuration to use
  2. Which environment to deploy to

For example:

mb eucalyptus bootstrap bootstrap.json -e edge-nightly -v

Standard
Eucalyptus, QA

Testing Riak CS with Eucalyptus

EUCA+RIAK-CS

Introduction

One of the beautiful things about working with IaaS is the disposable nature of instances. If they are not behaving properly due to a bug or have been misconfigured for some reason, instances can be terminated and rebuilt with more ease than debugging a long lived and churned through Linux system. As a quality engineer, this dispensability has become invaluable in testing and developing new tools without needing to baby physical or virtual machines.

One of the projects I have been working on lately is an easy deployment of Riak CS into the cloud in order to quickly and repeatedly test the object storage integration provided by Eucalyptus in the 4.0 release. Riak CS is a scalable and distributed object store that provides an S3 interface for managing objects and buckets.

Before testing the Eucalyptus orchestration of Riak CS (or any tool/backend/service that Euca supports for that matter), it is important to understand the basic activities that Eucalyptus will be performing on behalf of the user. Thankfully, Neil Soman wrote a great blog post about how our Riak CS integration is designed.

In this model  we can see that we require:

  1. A multi-node Riak CS cluster
  2. A load balancer
  3. A machine to run the Eucalyptus Object Storage Gateway (OSG)

This topology is extremely simple to deploy in Eucalyptus 3.4 using our ELB and by using Vagrant to deploy our Riak CS cluster. Here’ s how to get your groove on.

Prerequisites

  1. CentOS 6 image loaded into your cloud
  2. Keypair imported or created in the cloud
  3. Security group authorized for port 8080,8000 and 22
  4. Install Vagrant

Deploy Riak CS

In order to deploy Riak CS in our cloud we will use Vagrant+Chef+Berkshelf as follows:

  1. Install Vagrant plugins using the following commands:
    • vagrant plugin install vagrant-berkshelf
      vagrant plugin install vagrant-omnibus
      vagrant plugin install vagrant-aws
  2. Import the dummy vagrant box necessary to use vagrant-aws:
    • vagrant box add centos dummy.box
  3. Clone the following repository
    • git clone https://github.com/viglesiasce/vagrant-riak-cs-cluster.git
  4. Edit the following items in the Vagrantfile to reflect the pre-requisites above and to point to your target cloud
    • aws.access_key_id
    • aws.secret_access_key
    • aws.keypair_name
    • aws.ami
    • override.ssh.private_key_path
    • aws.security_groups
    • aws.endpoint
  5. Set the number of nodes to deploy at the top of the Vagrantfile:
  6.  Once the cloud options are set start the Vagrant “up” process which will deploy the Riak CS nodes and Stanchion:
    • RIAK_CS_CREATE_ADMIN_USER=1 vagrant up --provider=aws
  7. Once Vagrant is complete, login to the first Riak CS node to get its private hostname:
    • vagrant ssh riak1 -c "curl http://169.254.169.254/latest/meta-data/local-hostname"
  8. Join each node to the first that was deployed. For example, to join the second node to the cluster I would run:
    • vagrant ssh riak2 -c "riak-admin cluster join riak@<riak1-private-hostname>"
      vagrant ssh riak2 -c "riak-admin cluster plan; riak-admin cluster commit"

In order to get your access and secret keys login to http://riak1-public-ip:8000

Load Balance Your Riak CS Cluster

  1. Create an ELB with the following command:
    • eulb-create-lb -z <AZ-of-your-riak-nodes> -l "lb-port=80, protocol=TCP, instance-port=8080,instance-protocol=TCP" RiakCS
  2. The command above will return you the DNS name that you will use as the endpoint for the “objectstorage.s3provider.s3endpoint” property when setting up the OSG. From the sample output below we would use “RiakCS-229524229045.lb.home”
    • DNS_NAME        RiakCS-229524229045.lb.home
  3. Register your Riak CS nodes with that load balancer:
    • eulb-register-instances-with-lb --instances <instance-id-1>,<instance-id-2> RiakCS

You have now successfully deployed a Riak CS cluster. You can stop here if you’d like but the real fun starts when you add IAM, ACL, versioning, multipart upload, and bucket lifecycle support to the mix using the Eucalyptus OSG.

True enthusiasts continue below.

Install and Configure the Eucalyptus OSG Tech Preview

  1. Spin up another CentOS 6 instance in the same security group as used above
  2. Follow the instructions found here to finish  the OSG installation and configuration, remember to use the DNS name returned in step 1 from above as the s3endpoint:
Standard
Eucalyptus

Using Aminator with Eucalyptus

Introduction to Aminator

The Netflix Cloud Platform has shown how a large scale system can be deployed in a public cloud and maintain an extreme level of performance and reliability. As Adrian Cockcroft has said in the past, Netflix focused on having functional and scalable code rather than worrying immediately about how to make it portable.  At Eucalyptus we have been working over the past year on making sure that as many of the NetflixOSS projects that interact directly with the cloud can be used on Eucalyptus. This level of portability means that anyone with even 1 single linux box can use their system as a test bed for deploying NetflixOSS more broadly on a public cloud. So far at Eucalyptus we have working versions of Asgard, Simian Army, Aminator, and Edda. These tools are cornerstones for app deployment and monitoring with the NetflixOSS stack. In this post I will show how to use Aminator with your Eucalyptus cloud.

Aminator is a tool created by Netflix to populate and catalog application images that are the building blocks for any service infrastructure you can dream up. Aminator works by taking a “Foundation AMI” and mounting a snapshot from it in order to provision your application. It does this by mounting a volume, created from an image snapshot, to a running instance then performing a chroot that runs provisioners such as Chef or Ansible. Once the provisioning step is complete the volume is snapped and registered as an AMI. Aminator doesn’t stop there however. It also creates tags of the snapshot and the AMI so that they can be easily identified. Some of the information included in the tags is:

* Description
* Owner
* Package version info
* Name of application
* Base AMI

Having this information allows an Aminator user to trace back the history of how one of their applications was deployed, also pinning the deployment to a particular person.

Some of the benefits of using Aminator for app deployment include:

  •  Ensuring exact and dependable recovery of previous application stack (including dependencies and software)
  •  Allows applications to be deployed in AutoScaling groups as each AMI is completely self contained version of the application
  • Ensures application images are tagged with appropriate meta data for traceability
  • Allows traceability of ownership of images (since Netflix uses one large AWS account)

With the addition of Eucalyptus cloud to deploy on you can enjoy the following:

  • An internal test and development platform for NetflixOSS
  • Gives application developers an easy way to catalog, build, and deploy their test applications
  • Ensures a repeatable process is in place for creating an image that will eventually go into production
  • Test changes to an image quickly/cheaply on local private infrastructure before deploying into production

Using Aminator in Eucalyptus

In order to run Aminator we will first need to build our Aminator instance (which will also be the AMI we use as the “Foundation AMI”).

  1. Download the Ubuntu Precise QCOW disk image to a machine that has the qemu-img tool
  2. Convert the QCOW image to RAW format using the following command:
    • qemu-img convert -O raw ubuntu-12.04-server-cloudimg-amd64-disk1.img ubuntu-12.04-server-cloudimg-amd64-disk1.raw
  3. Once the image is converted start up an instance so we can create our “Foundation AMI”
  4. After the instance is booted copy the raw image to the instances ephemeral storage
  5. Attach a 2G volume to the instance
  6. Copy the disk file to the volume using dd:
    • dd if=ubuntu-12.04-server-cloudimg-amd64-disk1.raw of=/dev/vdb
  7. Create a snapshot from your volume
  8. Register the snapshot as an image in your cloud. Remember this AMI ID as it will be what we pass to Aminator in later steps.
  9. Run an instance from the newly created image and log into it

In these steps we have now created the base image for future application deployments and created the instance where we will run our Aminator tasks. Next up we will install Aminator and the Eucalyptus plugin:

  1. Clone the Aminator repository
    • git clone https://github.com/Netflix/aminator.git
  2. Edit the aminator/default_conf/environments.yml file and add the following block:
    • euca_apt_linux:
          cloud: euca
          distro: debian
          provisioner: apt
          volume: virtio
          blockdevice: virtio
          finalizer: tagging_ebs_euca
  3. Run the setup script twice from inside the aminator directory
    • cd aminator;python setup.py install; python setup.py install
  4. Now clone the eucalyptus-cloud Aminator plugin and install it
  5. Now that you have all the dependencies lets run an amination to install and label an Apache web server image:
    • sudo aminate -e euca_apt_linux –ec2-endpoint <clc-ip> -B emi-CD544111 apache2

In the above command we have told Aminator a few things about what we want to do:

  1. -e: Use the Eucalyptus environment for provisioning a Linux machine with APT
  2. –ec2-endpoint: Use this IP to connect to the Eucalyptus cloud
  3. -B: Use this AMI (the one we registered in the steps above) as the base image
  4. apache2: Install the apache2 package and tag the appropriate version information onto the snapshot and image

After a few minutes Aminator should complete, letting you know which AMI it has registered for you.

Enjoy your new application deployment tool!

Standard
Eucalyptus, QA

Extracting Info From Euca’s Logs

logstashIntroduction

Throughout my tenure as a Quality Engineer, I have had a love/hate relationship with logs. On one hand, they can be my proof that a problem is occurring and possibly the key to tracking down a fix. On the other hand, they can be an endless stream of seemingly unintelligible information. In debugging a distributed system, such as Eucalyptus, logging can be your only hope in tracing down issues with operations that require the coordination of many components.

Logs are generally presented to users by applications as flat text files that rotate their contents over time in order to bound the amount of space they will take up on the filesystem. Gathering information from these files often involves terminal windows, tail, less, and timestamp correlation. The process of manually aggregating, analyzing and correlating logs can be extremely taxing on the eyes and brain. Having a centralized logging mechanism is a great leap forward in streamlining the debug process but still leaves flat text files around for system administrators or testers to analyze for valuable information.

A month or so ago I sought out to reinvigorate my relationship with logs by making them sexy again. I looked around at the various open source and proprietary tools on the market and decided to give Logstash a shot at teaching me something new about Eucalyptus through its logs. The “getting started” links I found on the docs page presented a quick and easy way to see what LogStash could do for my use case, namely ingesting and indexing logs sent from rsyslog. Once I got some logs to appear in the ElasticSearch backend, I got a bit giddy as I was now able to search and filter the logs through an API. But alas! I was still looking at text on a freaking black and green screen. BORING! There had to be a better way to visualize this data.

I looked around a bit and found Kibana. This beautiful frontend to ElasticSearch gives you a simple and clean interface for creating/saving dashboards that reflect interesting information from your logs. Within minutes of installing Kibana, I had a personalized dashboard setup that was showing me the following statistics from my Eucalyptus install that was undergoing a stress test:

  • Instances run
  • Instances terminated
  • Volumes created
  • Volumes deleted

I had proven that there was value in using Logstash and it was not complicated to setup or use. I then began to use other dashboards, filters, and search terms to look for anomalous patterns in the log messages. This type of analysis resulted in a couple of issues being opened that I would not have found looking at one screen of text at a time.

Below I will outline the steps to begin your own Logstash journey with Eucalyptus or any other system/application that logs to a filesystem on a Linux box.

Installation

Installing Logstash

  1. Install packages
    • On Ubuntu: 
      apt-get install default-jre git apache2 ntp
    • On CentOS:
      yum install java-1.7.0-openjdk.x86_64 git httpd ntp
  2. Set proper timezone
    1. Ubuntu
    2. CentOS
  3. Download Logstash
    • wget https://logstash.objects.dreamhost.com/release/logstash-1.1.13-flatjar.jar -O logstash.jar
  4. Create LogStash config file for rsyslog input. Create and edit a file named logstash.conf
    • input {  syslog {    type => syslog    port => 5544  }}
      output {  elasticsearch { embedded => true } }
  5. Run logstash JAR
    • nohup java -jar logstash.jar agent -f logstash.conf &
  6. Configure rsyslog on Eucalyptus components by adding the following to the /etc/rsyslog.conf file and replacing <your-logstash-ip>
    • $ModLoad imfile   # Load the imfile input module$ModLoad imklog   # for reading kernel log messages
      $ModLoad imuxsock # for reading local syslog messages
      $InputFileName /var/log/eucalyptus/cloud-output.log
      $InputFileTag clc-sc-log:
      $InputFileStateFile clc-sc-log
      $InputRunFileMonitor
      $InputFileName /var/log/eucalyptus/cc.log
      $InputFileTag cc-log:
      $InputFileStateFile cc-log
      $InputRunFileMonitor
      *.* @@<your-logstash-ip>:5544
  7. Restart rsyslog
    • service rsyslog restart

Installing Kibana 3

  1. Clone the repository from GitHub
    • git clone https://github.com/elasticsearch/kibana.git
  2. Edit the kibana/config.js file and set the elasticsearch line to:
    • elasticsearch:    "http://<your-logstash-public-ip>:9200", 
  3. Copy the Kibana repository to your web server directory
    • CentOS:
      mv kibana/* /var/www/html/; service httpd start
    • Ubuntu:
      mv kibana/* /var/www/

Point your browser to http://<your-logstash-public-ip&gt; and you should be presented with the Kibana interface. Kibana is not specifically a frontend for Logstash but rather a frontend to any ElasticSearch installation. Kibana does provide a default Logstash dashboard as a starting point for you customizations:  http://<your-logstash-public-ip>/index.html#/dashboard/file/logstash.json

Standard
Eucalyptus

Getting Started with EucaLobo

Initial Setup

In my previous post, I described the story behind EucaLobo, a graphical interface for managing workloads on AWS and Eucalyptus clouds through a <cliche>single pane of glass</cliche>. The tool is built using Javascript and the XUL framework allowing it to be used on Linux, Windows, and Mac for the following APIs:

  • EC2
  • EBS
  • S3
  • IAM
  • CloudWatch
  • AutoScaling
  • Elastic Load Balancing

To get started download the binary for your platform:

Once installation is complete and EucaLobo starts for the first time you will be prompted to enter an endpoint. My esteemed colleague Tony Beckham has created a great intro video showing how to create and edit credentials and endpoints. The default values have been set to the Eucalyptus Community Cloud, a free and easy way to get started using Eucalyptus and clouds in general. This is a great resource for users who want to get a feel for Eucalyptus without an upfront hardware investment.

Enter the following details if you have your own cloud or would like to use AWS:

After entering an endpoint, the next modal dialog will request that you enter your credentials:

  • Name: Alias for these credentials
  • Access Key
  • Secret Key
  • Default Endpoint: Endpoint to use when these credentials are activated
  • Security Token: Unnecessary for most operations

Any number of endpoints and credentials can be added which makes EucaLobo ideal for users who leverage multiple clouds (both public and private). Once you have loaded up at least one endpoint and credential set, you need to:

  1. Go to the “Manage Credentials” tab
  2. Select a credential in the top pane
  3. Click the “Activate” button

You are now ready to start poking around the services available through EucaLobo. All services are listed on the left pane of the interface. Clicking on the name of the tabs will take you to the implementation of that functionality. The ElasticWolf team did a great job of making an intuitive and simple interface to navigate. As an enhancement, which I hope to get upstream soon, I have added labels to all buttons in the UI so that it is clear which operations will be executed.

Cool Features

Portability

ElasticWolf leverages the XUL framework which enables developers to write their application once and deploy it on Mac/Linux/Windows or any platform that supports Firefox. This level of portability is great to cover a large number of users with minimal effort. So far I have not found any bugs that are platform specific.

Multi-cloud

EucaLobo makes it easy to quickly change endpoints and credentials. My common use cases for this feature are:

  • Switching only endpoints – Switching regions in AWS
  • Switching both endpoint+credentials: – Verifying Eucalyptus behavior after testing in the same interface as AWS
  • Switching only credentials – Use different users to validate IAM behavior

multi-cloud

IAM Canned policies

One of the great workflows inherited from ElasticWolf is the ability to use pre-canned policies when associating a policy to users and groups.

canned-policy

Security features

You may be thinking that adding cloud credentials to an application and leaving it open on your desktop is too risky. You would be absolutely correct. To combat this risk, you can set an inactivity timer that will either exit the application or require the user to enter a preset password. The granularity of the timer can be set to as low as 1 minute.

security

S3 advanced features

One of the most powerful features in the S3 API is the ability to lock down (or open up) S3 entities (objects and buckets) using an ACL policy language. Unfortunately, the S3 ACL API is not the most user friendly. With the ACL implementation in EucaLobo, you can choose to share a file publically or share with only 1 or more individual users.

s3-acls

CloudWatch Graphs

The reason I began my efforts to get ElasticWolf working with Eucalyptus was in order to use it as an interface to the newly developed CloudWatch API in Eucalyptus. EucaLobo makes it extremely easy to visualize the usage of each of your instance, volumes, load balancers, and AutoScaling groups.

cloudwatch

Conclusion

EucaLobo has been extremely useful for me during the testing of Eucalyptus 3.3, as well as for managing my home private cloud and AWS accounts. I hope that others can find it as useful and useable as I have. With what I have learned during the development of EucaLobo, I hope to refork ElasticWolf  so that I can make a smaller patch upstream for enabling Eucalyptus Cloud support.

Please dont hesitate to provide feedback in the form of comments on this blog, on Github as issues, or on IRC at the #eucalyptus-qa channel of Freenode. As always pull requests are welcome: https://github.com/viglesiasce/EucaLobo

Standard
Eucalyptus

The Journey to EucaLobo

The 3.3.0 feature barrage

As a quality engineer it is always useful to have an at-a-glance view of the state of your system under test. Unfotunately, having reliable graphical tools is not always possible during testing phases as the UI is often trailing the development of core features. During the 3.3.0 release, the Eucalyptus development team added an incredible amount of API calls to its already large catalog of AWS compatible operations:

  • Elastic Load Balancing
  • Autoscaling
  • CloudWatch
  • Resource Tagging
  • Filtering
  • Maintenance Mode
  • Block Device Mappings

As a result of this onslaught of new service functionality from developers the UI and QA teams had their work cut out for them. The UI team had decided early on that they needed to make some architectural changes to the UI code, such as leveraging Backbone.js and Rivets. This meant they would only be able to cover the newly added resource tagging and filtering services within the 3.3.0 timeframe. Unfortunately, the UI was not the only client tool that needed to implement new services as Euca2ools 2.x was also lacking support for ELB, CloudWatch, and Autoscaling. As we split up the services amongst the quality engineers it became apparent that we had an uphill battle ahead and would need every advantage we could get. I took the lead for the CloudWatch service and began my research as soon as the feature had been committed to the release. In reading about and using the AWS version of CloudWatch it became clear that the service basically boiled down to:

  1. Putting in time series data
  2. Retrieving statistics on that data over a given interval at a set periodicity

Having worked with time series data before, I knew that without a way to visualize it I would be seriously hindering my ability to verify resulting metrics. I pulled out my handy recipe for Graphite and wrote a simple bash script that would grab a CloudWatch data set from a file and send it to my Graphite server using netcat. This worked as a quick proof of concept that we were storing the correct data and computing its statistics properly over longer periods. One of the major functionalities that is provided by the CloudWatch service is instance monitoring. This data allows users to make educated decisions about how and when to scale their applications. The realtime nature meant that I needed to be able to create arbitrary load patterns on instances and volumes and quickly map that back to CloudWatch data. It became clear that a bash script pulling from a set of text files was not going to be simple or flexible enough for the task.

Let the hacking begin

As I began looking around for CloudWatch visualizers, it was clear that not many people had attacked the problem, likely because the AWS Console implementation is solid. One project that almost immediately bubbled to the top, however, was ElasticWolf, the AWS console developed for use with GovCloud. This project had been around for a year or so and had managed to implement a graphical interface for every single service that AWS supported, including AutoScaling, which is still not found in the AWS Console. It seemed like it would not take much time to point the ElasticWolf interface at my Eucalyptus cloud, so I took a stab at the Javascript code that backs the XUL application and ended up with a working version within 24hrs.  This timeline from cloning my first repo to using EucaLobo as my daily driver is a testament to the API fidelity that Eucalyputs provides.  At that point, I had hardcoded many things in the code that made it no longer work with AWS, fortunately at the time hybrid functionality was irrelevant.  A few weeks later when I had a better idea of how the code was structured and how I could manipulate the UI elements, I was able to reimplement the credential and endpoint management such that it would allow hybrid functionality. This was another great advantage for our team in that we could now run the exact same operations on both AWS and Eucalyptus and compare the results through the same interface. ElasticWolf was also quite useful in defining the workflows that were common to the new services we had implemented. For example, its UI will ensure that there are launch configurations created before you attempt to create an autoscaling group. These types of guard rails allowed us to efficiently learn and master the new features with a low barrier to entry in order to deliver a high quality release within our schedule.

In my next post I will show how to get started with EucaLobo as well as highlight some of its features.

Standard