Eucalyptus

Using Comcast CMB for SQS and SNS on Eucalyptus

Introduction

As part of a service oriented infrastructure there comes a need to coordinate work between services. AWS provides a couple of services which allow for application components to communicate with each other and their users/administrators in a decoupled fashion.

The Simple Queue Service (SQS) is a mechanism for an applications producers to distribute work to their consumers in a scalable, reliable and fault tolerant way. The basic lifecycle in SQS is as follows:

  1. A queue is created
  2. Producers send arbitrary text messages into the queue
  3. Consumers are constantly listing the messages in a queue and when one is available they “check out” the work by reading the message
  4. Once the message is read a timer kicks that makes the message unreadable by other consumers for a certain period of time (called the visibility timeout).
  5. The consumer can then perform the necessary task described by the message and then delete the message from the queue
  6. If the consumer does not complete the task in time or fails for some other reason the message is made visible again in the queue and picked up by another consumer

One simple example of using this service would be for a Web application front end to take image conversion orders from a user and then throw the image conversion task into a queue that can then be serviced by a fleet of worker nodes that do the actual image processing (ie the compute heavy portion).

The Simple Notification Service (SNS) is a service that allows for the coordination of messages that have one or more recipient subscribing endpoints. In this service users create a topic and then other services and users can subscribe to the topic in order to receive notifications about its goings on. In this model the sender of the message does not have to know where messages are actually being sent but rather that all subscribers (ie people/apps who need the message) will receive the message in the form that they have requested. Subscriptions to topics can be made through various transport mechanisms:

  1. HTTP
  2. HTTPS
  3. SMS
  4. Email
  5. Email-json
  6. SQS

By publishing a message to a topic with multiple subscribers you can ensure that both applications and the people managing them are all on the same page.

Eucalyptus currently does not implement SQS and SNS but the folks over at Comcast have created an incredibly useful open source project that mirrors the APIs with absolutely incredible fidelity. Not only did they ensure that their API coverage was accurate and useful but they built the application stack on top of Cassandra and Redis making it not only horizontally scalable but extremely performant to boot. For more information: Comcast CMB.

Running CMB in your Eucalyptus cloud

In order to simplify the process of installing and bootstrapping CMB, I have created an image that you can install on your cloud with all the requisite services in place. All instructions here should be performed from your Eucalyptus CLC with your admin credentials sourced.

  1. Download the image and decompress it
    1. curl http://eucalyptus-images.s3.amazonaws.com/public/cmb.raw.xz > cmb.raw.xz
    2. xz -d cmb.raw.xz
  2. Install the image
    1. euca-install-image –virt hvm -i cmb.raw -r x86_64 -b CMB -n CMB
  3. Launch the image
    1. euca-run-instance -k <my-keypair> <emi-from-step-2>
  4. Once the image is launched login to the admin portal to create your first user and get your credentials
    1. Goto http://<instance-public-ip&gt;:6059/webui
    2. Login with: cns_internal/cns_internal
    3. Create a new user
    4. Take note of the Access and Secret keys for your new users
  5. Start using your new services with your favorite SDK

Example: Interacting with SQS using Boto

In the example below swap change the following variables to fit your environment:

  • cmb_host – Hostname or IP of your CMB server
  • access_key – Taken from step 4D above
  • secret_key – Taken from step 4D above
#!/usr/bin/python
from boto.sqs.regioninfo import SQSRegionInfo
from boto.sqs.connection import SQSConnection

cmb_host = 'instance-ip'
access_key = 'your-access-key-from-step-4D'
secret_key = 'your-secret-key-from-step-4D'
cmb_sqs_port = 6059

sqs_region = SQSRegionInfo(endpoint=cmb_host, name='home')
cmb_sqs = SQSConnection(aws_access_key_id=access_key, aws_secret_access_key=secret_key,
region=sqs_region, is_secure=False,
port=cmb_sqs_port)

queue = cmb_sqs.create_queue('test')
msg = queue.new_message('Hello World')
queue.write(msg)

all_queues = cmb_sqs.get_all_queues()
print 'Current queues: '  + str(all_queues)
for queue in all_queues:
    print 'Messages in queue: ' + str([msg.get_body() for msg in queue.get_messages()])
Standard
Uncategorized

Installing Apache Hadoop on Eucalyptus using Hortonworks Data Platform stack and Apache Ambari

Get your #hortonworks #hadoop running on #eucalyptus

shaon's blog

This post demonstrates Hadoop deployment on Eucalyptus using Apache Ambari on Horton Data Platform stack. Bits and pieces:

  1. Eucalyptus 4.0.0
  2. Hortonworks Data Platform (HDP) stack
  3. Apache Ambari
  4. 4 instance-store/EBS-backed instances
    1. 2 vcpus, 1024MB memory, 20GB disk space
    2. CentOS 6.5 base image

For this is demo we tried to use very minimum resources. Our Eucalyptus deployment topology looks like below, 1x (Cloud Controller + Walrus + Storage Controller + Cluster Controller) 1x (Node Controller + Object Storage Gateway) To meet our instance requirement, we changed the instance-type according to our need.

Preparation Run an instance of m1.xlarge or any other instance type that meets the above requirement. When the instance is running copy the keypair that is used to run this instance at .ssh/id_rsa, we will be using this same keypair for all the instances.

Run three more instance of same type with same keypair and image and copy their private IPs for later use. Add security…

View original post 200 more words

Standard
Eucalyptus

Install Eucalyptus 4.0 Using Motherbrain and Chef

 

Introduction

Installing distributed systems can be a tedious and time consuming process. Luckily there are many solutions for distributed configuration management available to the open source community. Over the past few months, I have been working on the Eucalyptus cookbook which allows for standardized deployments of Eucalyptus using Chef. This functionality has already been implemented in MicroQA using individual calls to Knife (the Chef command line interface) for each machine in the deployment. Orchestration of the deployment is rather static and thus only 3 topologies have been implemented as part of the deployment tab.

Last month, Riot Games released Motherbrain, their orchestration framework that allows flexible, repeatable, and scalable deployment of multi-tiered applications. Their approach to the deployment roll out problem is simple and understandable. You configure manifests that define how your application components are split up then define the order in which they should be deployed.

For example in the case of Eucalyptus we have cluster, node, and frontend components. Each component is a set of recipes from the Eucalyptus cookbook. Once we have recipes mapped to components we need to define the order in which these components should be rolled out in the “stack order” section of our Motherbrain manifest:

stack_order do
bootstrap ‘cloud::full’
bootstrap ‘cloud::default’
bootstrap ‘cloud::frontend’
bootstrap ‘cluster::default’
bootstrap ‘cluster::cluster-controller’
bootstrap ‘cluster::storage-controller’
bootstrap ‘cloud::user-facing’
bootstrap ‘cloud::walrus’
bootstrap ‘cloud::user-console’
bootstrap ‘node::default’
bootstrap ‘cloud::configure’
bootstrap ‘nuke::default’
end

Once we have the components split up and ordered we need to define our topology. This can we done with another JSON formatted manifest like so:

{“nodes”: [
{ “groups”: [“cloud::frontend”, “cloud::configure”],
“hosts”: [“10.0.1.185”]
},
{ “groups”: [“cluster::default”],
“hosts”: [“10.0.1.186”]
},
{ “groups”: [“node::default”],
“hosts”: [“10.0.1.187”, “10.0.1.181”]
}]

}

With this information, Motherbrain allows you to create arbitrary topologies of your distributed system with repeatability and scalability taken care of. Repeatability comes from using Chef recipes and the scalability is derived from the nodes in each tier being deployed in parallel. In Eucalyptus terms, this means that no matter how many Node Controllers you’d like to deploy to your cluster, they system will come up in almost constant time. In order to tweak the configuration you can deploy your stack into a properly parameterized Chef environment.
Now that the concept has been laid out, lets get to business building our cluster from the 4.0 nightlies.

Installing prerequisites

I have created a script to install and configure Motherbrain and Chef that should work for Enterprise Linux or Mac OSX:

sh <(curl -s https://gist.githubusercontent.com/viglesiasce/9734682/raw/install_motherbrain.sh)

If you’d like to do the steps manually you can:

  1. Install ruby 2.0.0
  2. Install gems
    1. chef
    2. motherbrain
    3. chef-zero
  3. Get cookbooks and dependencies
    1. eucalyptus – https://github.com/eucalyptus/eucalyptus-cookbook
    2. ntp – https://github.com/opscode-cookbooks/ntp.git
    3. selinux – https://github.com/opscode-cookbooks/selinux.git
    4. yum – https://github.com/opscode-cookbooks/yum.git
  4. Upload all cookbooks to your Chef server
  5. Configure Motherbrain
    1. mb configure

Customizing your deployment

  1. Go into the Eucalyptus cookbook directory (~/chef-repo/cookbooks/eucalyptus)
  2. Edit the bootstrap.json file to match your deployment topology
    1. Ensure at least 1 IP/Machine for each component
    2. Same IP can be used for all machines (Cloud-in-a-box)
  3. Edit the environment file in ~/chef-repo/cookbooks/eucalyptus/environments/edge-nightly.json
    1. Change the topology configuration to match what you have defined in the bootstrap.json file
    2. Change the network config to match your Eucalyptus deployment
  4. Upload your environment to the Chef server
    1. knife environment from file environments/edge-nightly.json

Deploying your Eucalyptus Cloud

Now that we have defined our topology and network configuration we can deploy the cookbook using the Motherbrain command line interface by telling the tool:

  1. Which bootstrap configuration to use
  2. Which environment to deploy to

For example:

mb eucalyptus bootstrap bootstrap.json -e edge-nightly -v

Standard
Uncategorized

Pursuit of Quality at Eucalyptus

Overview of the extensive Eucalyptus quality toolchain.

shaon's blog

At Eucalyptus, if one thing we care about it is the quality of the product. Our goal is to deliver an AWS compatible software that just works. To ensure the highest level of quality we try to follow the optimum strategy possible in both Development and in QA.

We have adopted Agile Software Development model a while back and we love it. As a part of our strategy, there are several type of projects we open in Jira. After getting the green signal from Product Management, we get Epic and then after architectural discussion Story tickets are created with Sub-tasks for developers to work on. We also have Jira tickets for New Features which are basically similar to Story but comparatively smaller tasks and generally have no Sub-tasks. At this point one scrum master from each team works closely with the developers to keep track of the…

View original post 629 more words

Standard
Eucalyptus, QA

Testing Riak CS with Eucalyptus

EUCA+RIAK-CS

Introduction

One of the beautiful things about working with IaaS is the disposable nature of instances. If they are not behaving properly due to a bug or have been misconfigured for some reason, instances can be terminated and rebuilt with more ease than debugging a long lived and churned through Linux system. As a quality engineer, this dispensability has become invaluable in testing and developing new tools without needing to baby physical or virtual machines.

One of the projects I have been working on lately is an easy deployment of Riak CS into the cloud in order to quickly and repeatedly test the object storage integration provided by Eucalyptus in the 4.0 release. Riak CS is a scalable and distributed object store that provides an S3 interface for managing objects and buckets.

Before testing the Eucalyptus orchestration of Riak CS (or any tool/backend/service that Euca supports for that matter), it is important to understand the basic activities that Eucalyptus will be performing on behalf of the user. Thankfully, Neil Soman wrote a great blog post about how our Riak CS integration is designed.

In this model  we can see that we require:

  1. A multi-node Riak CS cluster
  2. A load balancer
  3. A machine to run the Eucalyptus Object Storage Gateway (OSG)

This topology is extremely simple to deploy in Eucalyptus 3.4 using our ELB and by using Vagrant to deploy our Riak CS cluster. Here’ s how to get your groove on.

Prerequisites

  1. CentOS 6 image loaded into your cloud
  2. Keypair imported or created in the cloud
  3. Security group authorized for port 8080,8000 and 22
  4. Install Vagrant

Deploy Riak CS

In order to deploy Riak CS in our cloud we will use Vagrant+Chef+Berkshelf as follows:

  1. Install Vagrant plugins using the following commands:
    • vagrant plugin install vagrant-berkshelf
      vagrant plugin install vagrant-omnibus
      vagrant plugin install vagrant-aws
  2. Import the dummy vagrant box necessary to use vagrant-aws:
    • vagrant box add centos dummy.box
  3. Clone the following repository
    • git clone https://github.com/viglesiasce/vagrant-riak-cs-cluster.git
  4. Edit the following items in the Vagrantfile to reflect the pre-requisites above and to point to your target cloud
    • aws.access_key_id
    • aws.secret_access_key
    • aws.keypair_name
    • aws.ami
    • override.ssh.private_key_path
    • aws.security_groups
    • aws.endpoint
  5. Set the number of nodes to deploy at the top of the Vagrantfile:
  6.  Once the cloud options are set start the Vagrant “up” process which will deploy the Riak CS nodes and Stanchion:
    • RIAK_CS_CREATE_ADMIN_USER=1 vagrant up --provider=aws
  7. Once Vagrant is complete, login to the first Riak CS node to get its private hostname:
    • vagrant ssh riak1 -c "curl http://169.254.169.254/latest/meta-data/local-hostname"
  8. Join each node to the first that was deployed. For example, to join the second node to the cluster I would run:
    • vagrant ssh riak2 -c "riak-admin cluster join riak@<riak1-private-hostname>"
      vagrant ssh riak2 -c "riak-admin cluster plan; riak-admin cluster commit"

In order to get your access and secret keys login to http://riak1-public-ip:8000

Load Balance Your Riak CS Cluster

  1. Create an ELB with the following command:
    • eulb-create-lb -z <AZ-of-your-riak-nodes> -l "lb-port=80, protocol=TCP, instance-port=8080,instance-protocol=TCP" RiakCS
  2. The command above will return you the DNS name that you will use as the endpoint for the “objectstorage.s3provider.s3endpoint” property when setting up the OSG. From the sample output below we would use “RiakCS-229524229045.lb.home”
    • DNS_NAME        RiakCS-229524229045.lb.home
  3. Register your Riak CS nodes with that load balancer:
    • eulb-register-instances-with-lb --instances <instance-id-1>,<instance-id-2> RiakCS

You have now successfully deployed a Riak CS cluster. You can stop here if you’d like but the real fun starts when you add IAM, ACL, versioning, multipart upload, and bucket lifecycle support to the mix using the Eucalyptus OSG.

True enthusiasts continue below.

Install and Configure the Eucalyptus OSG Tech Preview

  1. Spin up another CentOS 6 instance in the same security group as used above
  2. Follow the instructions found here to finish  the OSG installation and configuration, remember to use the DNS name returned in step 1 from above as the s3endpoint:
Standard
Uncategorized

Hobos and Vagrants and Quality

Flattering blog post by the homie gregdek. Go get your micro-qa on!

Greg DeKoenigsberg Speaks

I don’t know how many people have actually met Vic Iglesias, our Quality Hobo.  Here’s what he looks like in his natural habitat:

It’s really super important not to make Quality Hobo angry. Let me assure you from personal experience: no one wants that.

Here are some things that make Quality Hobo angry (and that you should, therefore, avoid):

  • Stealing Quality Hobo’s cigarettes, electronic or otherwise.
  • Remarking upon Quality Hobo’s resemblance to a certain celebrity.
  • Wasting Quality Hobo’s time with questions about which test cases are most recent.
  • Wasting Quality Hobo’s time by writing tests that don’t integrate into Eutester.
  • Wasting Quality Hobo’s time with questions on how to build your QA environment.
  • Wasting Quality Hobo’s time in any way at all.

Here’s what I’m saying: it was only a matter of time before our Hobo got mixed up with a Vagrant.

Vagrant is incredible, and Mitchell Hashimoto is incredible for creating it.  It’s the…

View original post 391 more words

Standard
Eucalyptus

Using Aminator with Eucalyptus

Introduction to Aminator

The Netflix Cloud Platform has shown how a large scale system can be deployed in a public cloud and maintain an extreme level of performance and reliability. As Adrian Cockcroft has said in the past, Netflix focused on having functional and scalable code rather than worrying immediately about how to make it portable.  At Eucalyptus we have been working over the past year on making sure that as many of the NetflixOSS projects that interact directly with the cloud can be used on Eucalyptus. This level of portability means that anyone with even 1 single linux box can use their system as a test bed for deploying NetflixOSS more broadly on a public cloud. So far at Eucalyptus we have working versions of Asgard, Simian Army, Aminator, and Edda. These tools are cornerstones for app deployment and monitoring with the NetflixOSS stack. In this post I will show how to use Aminator with your Eucalyptus cloud.

Aminator is a tool created by Netflix to populate and catalog application images that are the building blocks for any service infrastructure you can dream up. Aminator works by taking a “Foundation AMI” and mounting a snapshot from it in order to provision your application. It does this by mounting a volume, created from an image snapshot, to a running instance then performing a chroot that runs provisioners such as Chef or Ansible. Once the provisioning step is complete the volume is snapped and registered as an AMI. Aminator doesn’t stop there however. It also creates tags of the snapshot and the AMI so that they can be easily identified. Some of the information included in the tags is:

* Description
* Owner
* Package version info
* Name of application
* Base AMI

Having this information allows an Aminator user to trace back the history of how one of their applications was deployed, also pinning the deployment to a particular person.

Some of the benefits of using Aminator for app deployment include:

  •  Ensuring exact and dependable recovery of previous application stack (including dependencies and software)
  •  Allows applications to be deployed in AutoScaling groups as each AMI is completely self contained version of the application
  • Ensures application images are tagged with appropriate meta data for traceability
  • Allows traceability of ownership of images (since Netflix uses one large AWS account)

With the addition of Eucalyptus cloud to deploy on you can enjoy the following:

  • An internal test and development platform for NetflixOSS
  • Gives application developers an easy way to catalog, build, and deploy their test applications
  • Ensures a repeatable process is in place for creating an image that will eventually go into production
  • Test changes to an image quickly/cheaply on local private infrastructure before deploying into production

Using Aminator in Eucalyptus

In order to run Aminator we will first need to build our Aminator instance (which will also be the AMI we use as the “Foundation AMI”).

  1. Download the Ubuntu Precise QCOW disk image to a machine that has the qemu-img tool
  2. Convert the QCOW image to RAW format using the following command:
    • qemu-img convert -O raw ubuntu-12.04-server-cloudimg-amd64-disk1.img ubuntu-12.04-server-cloudimg-amd64-disk1.raw
  3. Once the image is converted start up an instance so we can create our “Foundation AMI”
  4. After the instance is booted copy the raw image to the instances ephemeral storage
  5. Attach a 2G volume to the instance
  6. Copy the disk file to the volume using dd:
    • dd if=ubuntu-12.04-server-cloudimg-amd64-disk1.raw of=/dev/vdb
  7. Create a snapshot from your volume
  8. Register the snapshot as an image in your cloud. Remember this AMI ID as it will be what we pass to Aminator in later steps.
  9. Run an instance from the newly created image and log into it

In these steps we have now created the base image for future application deployments and created the instance where we will run our Aminator tasks. Next up we will install Aminator and the Eucalyptus plugin:

  1. Clone the Aminator repository
    • git clone https://github.com/Netflix/aminator.git
  2. Edit the aminator/default_conf/environments.yml file and add the following block:
    • euca_apt_linux:
          cloud: euca
          distro: debian
          provisioner: apt
          volume: virtio
          blockdevice: virtio
          finalizer: tagging_ebs_euca
  3. Run the setup script twice from inside the aminator directory
    • cd aminator;python setup.py install; python setup.py install
  4. Now clone the eucalyptus-cloud Aminator plugin and install it
  5. Now that you have all the dependencies lets run an amination to install and label an Apache web server image:
    • sudo aminate -e euca_apt_linux –ec2-endpoint <clc-ip> -B emi-CD544111 apache2

In the above command we have told Aminator a few things about what we want to do:

  1. -e: Use the Eucalyptus environment for provisioning a Linux machine with APT
  2. –ec2-endpoint: Use this IP to connect to the Eucalyptus cloud
  3. -B: Use this AMI (the one we registered in the steps above) as the base image
  4. apache2: Install the apache2 package and tag the appropriate version information onto the snapshot and image

After a few minutes Aminator should complete, letting you know which AMI it has registered for you.

Enjoy your new application deployment tool!

Standard