Eucalyptus

Using Scalr for Automation of your Eucalyptus Cloud

Introduction

scalr_logo

 

Euca logo

I have been using Eucalyptus heavily (as a quality engineer it is my day to day) for the past 1.5 years. I know the ins and the outs of system and am constantly tracking new features and bug fixes that arrive. With this knowledge it makes me a prime candidate to find out how other pieces of the cloud story can integrate with Eucalyptus.

I run a small cloud at home that I use for development and testing of different software stacks. Some of the tools that I’ve learned to use and hack on since Ive turned up my cloud include Graphite, Gitorious, Jenkins, Testlink, Zenoss. The issue with getting most of these (and any) open-source tools running is that they often require a very particular  base OS and dependency versions in order to install cleanly. This makes Eucalyptus a great tool for figuring out the right/easiest way to deploy an open source tool as it allows users to create and save any particular stack they have created for later use. Another great part about using Eucalyptus as a development tool is that it allows any and all distros to be loaded into a cloud and available for use by many users. When I look into a new tool to use I can rest assured that I can find one of their supported distros in my Eucalyptus cloud. Currently I have images registered for Debian 5/6, Ubuntu 12.04/12.10/13.04, Fedora 17, CentOS 6, and begrudgingly Windows 7 (99% used to manage vCenter nodes yuck).

There are many ways to populate and provision application stacks in a cloudy model. One way would be to load all of the application stack onto an image then re-bundle it and save it off. Another approach, and the one that I rely on, is to populate base images that can then be provisioned by scripts in order to run the application of choice. In using this model I had come up with many different scripts/user-data to populate the images listed above with applications. My approach allowed for apps to be deployed easily but I ended up with lots of replicated code across the user-data that I was passing to instances. Another pain was that it required me to remember which user-data scripts belonged with which images. Although not impossible to turn up and down my apps, it was definitely cumbersome and certainly not push button.

Enter Scalr.

Scalr allows me to populate scripts into a common database that is shared across all images/clouds. Because Scalr allows multiple scripts to be run during instance turn up (and manually for that matter), I am now able to modularize my scripts to reduce the amount of code I need to maintain.  Another element of Scalr that makes my life easier is its ability to auto scale an application. For my purposes I am not scaling an app to more than 1 instance, but if I ever bring down an instance erroneously Scalr will automatically repopulate the instance and run the scripts necessary to reconstruct the app. The final efficiency I get out of using Scalr is that I can load up multiple clouds and use the same scripts against similar base images within various clouds. As a tester I bring up and down at least 1 private cloud a day, on top of using my AWS account for both testing and production use, so this may be the last feature but certainly not least in my mind.

Eucalyptus+Scalr is a developer/tester’s dream, so lets get down to business on how to get this solution running on your own cloud.

Install Scalr

  1. Run the script found here on a Ubuntu Precise 12.04 server or instance: https://gist.github.com/4527791
  2. You will be prompted a few times to enter the password listed in the script (default is yourpassword)
  3. Ensure that the default virtual host (scalr.local) can be resolved by either adding a DNS entry to the server or a host entry on your client. Default hostname can be changed on lines 4-5 in the script from step 1.
  4. Login to Scalr as admin/admin: http://scalr.local
  5. Click on Accounts in the top left corner and create a user account
  6. Click Settings->Core Settings, and set the “Event handler URL” at the bottom to the hostname you chose in step 3 (default is sclar.local)
  7. Logout and back in with your user credentials created in step 5

Scalrize Your Images

You need to install an agent, Scalarizr, on any  any images used to create server templates, or in Scalr parlance roles. After installing the Scalarizr agent you will register each of your images as a role. To install Scalarizr:

  1. Install the repository package as appropriate for your distro: Debian  RHEL/CentOS
  2. Install the Scalarizr package appropriate to the cloud that will be used. In our case “scalarizr-eucalyptus”
  3. Ensure that your image can properly resolve the “Event Handler URL” entered when installing Scalr
  4. Rebundle and register the image with Eucalyptus

Register Your Eucalyptus Cloud

Scalr works with many cloud providers including AWS. They were able to leverage a good amount of  client code  in order to support Eucalyptus. It would seem that the last time that the Eucalyptus integration was looked at was in the Euca 2.x timeframe so many things that Eucalyptus now has full support for with 3.1+ (EBS backed images for example) are not supported. Other missing functionality is support for keypairs (which can be patched using scripts in Scalr) and all instances are launched in the default group (not sure what the reason for this is).

In order to setup Scalr with your cloud credentials:

  1. In the top left of the navigation bar click “default”, the “Manage”
  2. Click “Actions” on the right, then “Configure”
  3. Set your timezone
  4. Click the Eucalyptus logo
  5. Click the green plus sign to add a new Eucalyptus cloud

sclar-add-cloud

Creating a Role

Scalr uses a paradigm for cloud automation that requires that cloud images be registered as Roles. These roles are then added to Farms in order to deploy an app. Each role can only appear once per Farm. Scalr allows you to catalog, version and deploy scripts using a templating mechanism where values can be set at the Role, Farm, or individual server level.

Some common role types are:

  1. Base Images
  2. Load Balancers
  3. Web Servers

In order to create a Role:

  1. Go to this page, replacing the hostname if necessary: http://scalr.local/#/roles/edit
  2. Once there, enter a name for the role, for example: Ubuntu Precise Base
  3. Click on the check box that represents what category of role this is, in our case we will check the checkbox for Base
  4. Click over to the Images tab and enter the information about the image you registered in the steps above
  5. Enter the image info including its machine image ID
  6. Click Add (left side) then Save (bottom center)

Scalr - Roles

Adding your scripts

One of the most powerful parts about Scalr is the ability to write and reuse templated scripts. Scalr also allows you to share, fork, and version your scripts. Scripts can be added to Roles, Farms or individual servers for execution at boot, termination or manually during any part of an instances’ lifecycle. Creating, managing, and deploying scripts allows you to work around some shortcomings of the current Scalr/Eucalyptus integration like only being able to use the default security-group and no keypair being passed to instances launched through Scalr.

  1. Go to http://scalr.local/#/scripts/view in order to add some basic scripts 
  2. Click the green plus sign on the right of the Scalr Web Console
  3. When adding scripts you will need to give them a name and a description along with the actual code
  4. The first script we will add will be called “SSH Key Inject” and ensure that our SSH key can be added to an instance: 
    #!/bin/bash
    echo %ssh_key% > /root/.ssh/authorized_keys
    
  5. The next script we will add will install Graphite, a scalable realtime graphing application and can be found here

You will notice that the script added in step 2 uses a wild card parameter, %ssh_key%, that can be configured at script run time, role provisioning or farm launch time. Once we’ve created our scripts its time to create our farm which will pair our scripts with our images, or Roles, so that we can run and terminate our application at will.

Scalr - Scripts

Creating a Farm

Farms are collections of roles that constitute a single deployment. All servers in a farm can be turned up and down in unison in order to deploy an application. Individual roles within a farm can be autoscaled. When Scalr notices less than a certain threshold of servers in a particular role, it will automatically launch more servers. As an example Farm, I will be showing how to deploy Graphite (Scalable Realtime Graphing).

  1. Add a role using the procedure above that references a Scalarized Precise Base Image 
  2. Click over to the Farms pane in the Scalr web interface
  3. Click the green plus sign on the right of the Scalr Web Console
  4. Enter a name for this farm, in our case: Graphite
  5. Click over to the Roles tab and then click on Roles Library
  6. Click the plus sign next to Base Images
  7. Click on the icon for the Precise Base Image, then click the green plus sign.

Now that we have added the Base Image role to the Farm we will need to add scripts to the Role that run once the instance is up and running.

  1. Click on the icon that has now appeared towards the top of the page in order to configure what scripts we will run on this Role in the Farm
  2. On the bottom left click Scripting. In this configuration page we can choose which scripts to run, in which order and at what times
  3. Under the “When” dropdown click “Host Up”, then under the “execute” drop down choose the “SSH Key Inject” script we added in the steps from above. Once those two options have been chosen, click the green plus sign
  4.   Click on the row that has shown up for that script
  5. On the right side of the pane:
    1. Ensure that “Where” is set to “All instances of this role”
    2. Under the parameters section at the bottom of the pane, set which public key we want to inject for this role
  6. Add our second script “Install Graphite” at “Host Up”, ensure that “Where” is set to “All instances of this role”
  7. Click save at the bottom of the interface

Once we have defined what roles and scripts are paired up to make our application we can launch our Farm.

  1. Go to the Farms interface: http://scalr.local/#/farms
  2. Click Actions on the right side page
  3. Click Launch

Scalr will now launch your image and run the desired scripts to build out the application. You can use this interface to turn up and down your applications as necessary. If Scalr notices that your app is no longer running on the cloud it will be automatically relaunched for you. I have used this feature many times to help me with disaster recovery. Once I have an application running properly in my cloud I ensure that I can terminate any individual instances and my Scalr configuration is setup to properly rebuild the app.

scalr-farms

About these ads
Standard

20 thoughts on “Using Scalr for Automation of your Eucalyptus Cloud

  1. Excellent post! Seems like we have one more happy Scalr user – welcome to our community!

    Small typo: “Logout and back in with your user credentials created in step 4″. It’s step 5, actually.

    • Fixed! Thanks for the feedback.

      Can’t wait to see this integration get better and better. Do you accept pull requests? Do you have any developer docs to help get me started?

  2. lara says:

    hi sir
    i follow the your steps in install scalr and register eucalyptu cloud with it but i have problem in install scalarizer in image (i installed the repository package i am using centos 6.3)

    here is output
    Loading mirror speeds from cached hostfile
    * base: mirror.vit.com.tr
    * epel: mirror1.ku.ac.th
    * extras: mirror.vit.com.tr
    * updates: mirror.vit.com.tr
    Setting up Install Process
    No package scalarizr available.
    Error: Nothing to do

  3. lara says:

    hi
    i follow the above steps and create a farm but when i launch it the server status keep in pending launch , i guss maybe because scalarizer image cant resolve scalr.local ?
    which ip address should associate with scalr,local in /etc/hosts ? 127.0.0.1 or ip of machine in which i installed scalr in my case 192.168.1.100?

    is there any way to ensure connectivity between scalr server and my eucalyptus cloud?
    thank you in advance .

    • The entry in /etc/hosts should point to your Scalr server. Is your Scalr server running as an instance inside the cloud? If so try to autorize all ports to that security group as I am not sure what the minimum firewall rules are for a Scalr server.

  4. lara says:

    thank you viglesiasce
    no i installed scalr in a seperate machine not instance and its connected to the same lan of eucalyptus cloud i m using TP-LINK router cloud ip 192.168.1.104 and scalr server 192.168.1.100 and about instances i assaing range 192.168.1.120-192.168.1.160 for them .

    and of course i install scalaizer inside one of images that cloud used for running instance (i m using centos 6.3 image )

  5. lara says:

    Mr viglesiasce i am sure there is connection problem between cloud and scalr server just let me check with you register eucalyptus cloud after choose cloud type (in our case eucalyptus) scalr ask for enter eucalyptus cloud information as follow :

    http://paste.fedoraproject.org/4698/84438913/

    is there something wrong in entering my information ?

  6. lara says:

    thanks for reply

    is this part ok : X.509 Certificate file: euca2-admin-6eb71c39-cert.pem i am not sure which file i should put here so i put euca2-admin-6eb71c39-cert.pem

  7. lara says:

    i dont think there is problem with scalr connect eucalyptus cloud when i edit role and set placement and type i can see my cluster but i dont know why server stock in pending launch i ask scalr team but they say they dont support open source scalr that way i ask help from you and i hope i am not bothering you with my questions.

  8. Murali M says:

    I have installed scalr on a linux box and when i am trying to import a windows system running in the cloud i am getting this error

    KeyError: u’/dev/sda1/’

    eucalyptus version : 3.1

    can anybody help me out with this. i am struck in a big way.

    Thanks in advance.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s