Kubernetes

Policy-based Image Validation for Kubernetes with Anchore Engine

Introduction

As part of the journey to Kubernetes, you choose a set of Docker images that will house the dependencies needed to run your applications. You may also be using Docker images that were created by third parties to ship off-the-shelf software. Over the last year, the Kubernetes community has worked hard to make it possible to run almost any compute workload. You may now have operational tools such as monitoring and logging systems, as well as any number of databases and caching layers, running in your cluster alongside your application.

Each new use of Kubernetes requires that you vet the images that you will have running inside your cluster. Manual validation and introspection of images may be feasible, albeit tedious, to do for a very small number of images.  However, once you’ve outgrown that, you will need to set up automation to help ensure critical security vulnerabilities, unwanted dependency versions, and undesirable configurations don’t make their way into your clusters.

In the 1.9 release, Kubernetes added beta support for user configurable webhooks that it queries to determine whether a resource should be created. These ValidatingAdmissionWebhooks are a flexible way to dictate what configurations of resources you’d want to allow into your cluster.  To validate the security of your images, you can send each pod creation request to a webhook and return whether the image in the PodSpec adheres to your policies. For example, a naive implementation could allow pod creation only when images come from a set of whitelisted repositories and tags.

For more complicated image policies, you may need a more specialized tool that can gate your pod creation. Anchore Engine is “an open source project that provides a centralized service for inspection, analysis and certification of container images.” With its policy engine you can create and enforce rules for what you consider a production-ready image. As an example, an Anchore policy might check the following things about an image:

  1. No critical package OS vulnerabilities
  2. FROM always uses a tag
  3. Image runs as non-root by default
  4. Certain packages are not installed (ie openssh-server)

Below is a screenshot of these rules in the anchore.io policy editor:

In the next section, I’ll demonstrate how Anchore Engine can be integrated into your Kubernetes clusters to create a ValidatingWebhookConfiguration that runs all images through an Anchore policy check before allowing it into the cluster.

Mechanics of the image validation process

Infrastructure

The image validation process requires a few things to be deployed into your cluster before your policies can be enforced. Below is an architecture diagram of the components involved.

First, you’ll need to deploy Anchore Engine itself into the cluster. This can be achieved by using their Helm chart from the kubernetes/charts repository. Once installed, Anchore Engine is ready to accept requests for images to scan. Anchore doesn’t yet have a native API that Kubernetes can use to request validations. As such, we will create a small service inside of Kubernetes that can proxy requests from the ValidatingWebHookConfiguration.

The process of registering an admission server is relatively straightforward and can be accomplished with a single resource created in the Kubernetes API. Unfortunately, it takes more effort to create the server side process that will receive and respond to the admission request from Kubernetes. Thankfully, the good folks at Openshift have released a library that can be used as a base for creating admission servers. By leveraging the Generic Admission Server, we can reduce the amount of boilerplate in the server process and focus on the logic that we are interested in. In this case, we need to make a few requests to the Anchore Engine API and respond with a pass/fail result when we get back the result of Anchore’s policy check.

For instructions on setting this all up via a Helm chart, head over to the quick start instructions in the Kubernetes Anchore Image Validator repository.

Now that the server logic is handled, you can deploy it into your cluster and configure Kubernetes to start sending any requests to create Pods. With the infrastructure provisioned and ready, let’s take a look at the request path for an invocation to create a deployment in Kubernetes.

Request Flow

As an example, the following happens when a user or controller creates a Deployment:

  1. The Deployment controller creates a ReplicaSet which in turn attempts to create a Pod
  2. The configured ValidatingWebhook controller makes a request to the endpoint passing the PodSpec as the payload.
  3. The Anchore validation controller configured earlier will take that spec, pull out the image references from each container in the Pod and then ask Anchore Engine for a policy result using the Anchore API.
  4. The result is then sent back to Kubernetes.
  5. If the check fails for any of the images, Kubernetes will reject the Pod creation request.

Multi Cluster Policy Management

It is common to have multiple Kubernetes clusters with varying locations, purposes, or sizes. As the number of clusters grows, it can be hard to keep configurations in sync. In the case of the image policy configuration, you can leverage Anchore’s hosted service to serve as the source of truth for all of your clusters’ image policies.

Policy-based Image Validation for Kubernetes with Anchore Engine

Anchore allows you to create, edit, and delete your policies in their hosted user interface, then have the policies pulled down by each of your Anchore Engine services by turning on the policy sync feature. Once policy sync is enabled, Anchore Engine will periodically check with the hosted service to see if any updates to the active policy have been made. These may include new approved images. or the addition/removal of a rule.

What’s Next?

Head over to the Kubernetes Anchore Image Validator repository to check out the code or run the quick start to provision Anchore Engine and the validating web hook in your cluster.

Big thanks to Zach Hill from Anchore for his help in getting things wired up.

Standard
Uncategorized

Porting my brain from Python to Go

Over the past few months, I have made it a goal to get down and dirty with Go. I have been getting more and more involved with Kubernetes lately and feel like its time that I pick up its native language. For the last 6 years, I’ve had Python as my go-to language with Ruby sneaking in when the time is right. I really enjoy Python and feel extremely productive in it. In order to get to some level of comfort with Go I knew that I would have to take a multi-faceted approach to my learning.

Obviously, the first thing I did was just get it installed on my laptop. Being a Mac and Homebrew user, I simply ran: brew install golang. So now I had the toolchain installed and could compile/run things on my laptop. Unfortunately ‘Hello World’ was not going to get me to properly port my existing programming skills into using Go as a primary/native language. I knew enough about Go to know that I need to understand how and why it was built. Additionally, I had been in interpreted language land my whole life (Perl->Python->Ruby) so was taking another leap outside of just changing dialects. At this point I took a step back from just poking at source code to properly study up on the language.

I’ve had a Safari Books Online subscription for quite some time and have leveraged it heavily when learning new technologies. This case was no different. I picked up Programming in Go: Creating Applications for the 21st Century. I found this book to be at the right level for me, existing programming experience but new to Go. I also found that there was enough context as to how to do things the Go way such that I wouldn’t just be writing a bunch of Python in a different language. After a few flights browsing the concepts, I started to read it front to back. At the same time, I found a resource specific to my task at hand, Go for Python Programmers. This was a good way to see the code I was accustomed to writing and how I might write it differently in Go. As I continued reading and studying up, I made sure to pay particular attention when looking at Go code in the wild. The ask of myself was really to understand the code and the idioms, not just glance at it as pseudocode.

There were a few things that I needed some more clarification on after reading through my study materials. I was still confused about how packaging worked in practice. For clarification, my buddy and Go expert Evan Brown pointed me to the Go Build System repository. The thing that clicked for me here was the structure of having a repository of code that is related to each other, splitting out libraries into directories, then using the cmd directory in order to make the binaries that tied things together. This repo also has a great README that shows how they have organized their code. Thanks Go Build Team!

The next thing that I needed to hash out was how exactly I would use my Object Oriented penchant in Go. For this I decided to turn to Go by Example which had a wealth of simple example code for many of the concepts I was grappling with. Things started to click for me after looking at  structsmethods, and interfaces again through a slightly different lens.

Sweet! So now I understood (more or less) how the thing worked but I hadn’t built anything with it. The next phase was figuring out how the rubber met the road and having a concrete task to accomplish with my new tool.

I didn’t have any projects off the top of my head that I could start attacking but remembered that a few moons ago I had signed up for StarFighters, a platform that promised to provide a capture the flag style game that you could code against. I looked back through their site and noticed that their first game, StockFighter, had been released. StockFighter provides a REST API that players can code against in order to manipulate a faux stock market. I didn’t know anything about the stock market but figured this would be as good a task as any to get started. I played through the first few levels by creating a few one off binaries. Then on the harder levels I started to break out my code, create libraries, workers and all kinds of other pieces software to help me complete the tasks that  StockFighter was throwing at me. One huge help in getting me comfortable with creating this larger piece of software was that I had been using Pycharm with the Go plugin. This made code navigation, refactoring, testing and execution familiar.

Shit had gotten real. I was building a thing and feeling more comfortable with each level I played.

After my foray with StockFighter, I felt like I could use a different challenge. It turns out if you start asking around if people need some software built at Google, there will be plenty of people who want to take you up on the offer. The homie Preston was working on an IoT demonstration architecture and needed a widget that could ingest messages from PubSub and then store the data in BigTable. As he explained the project, I told him that it should take me no more than 4 hours to complete the task. I hadn’t use the Go SDKs at the time so I figured that would eat up most of my time. I sat down that afternoon and started the timer. This was the litmus test. I knew it. My brain knew it. My fingers knew it.

I made it happen in just under 5 hours which to me was a damn good effort as I knew that generally estimates are off by 2x. After doing a little cross-compiling magic, I was able to ship Preston a set of binaries for Mac, Linux and Windows that achieved his task.

I’m certainly not done with my journey but I’m happy to have had small successes that make me feel at home with Go.

 

Standard
Uncategorized

Building a container service with Mesos and Eucalyptus

  marathon-blog
    Over the past few months, I’ve been digging into what it means to work with a distributed container service. Inspired by Werner Vogel’s latest post about ECS, I decided to show an architecture for deploying a container service in Eucalyptus. As part of my investigations into containers, I have looked at the following platforms that provide the ability to manage container based services:
Each of these provide you with a symmetrical (all components run on all hosts) and scalable (hosts can be added after initial deployment) system for hosting your containerized workloads. They also include mechanisms for service discovery and load balancing. Deis and Flynn are both what I would call a “lightweight PaaS” akin to a private Heroku. Mesos, however, is a more flexible and open ended platform, which comes as a blessing and a curse. I was able to deploy many more applications in Mesos but it took me far longer to get a working platform up and running.
     Deis and Flynn are both “batteries included” type systems that once deployed allow you to immediately push your code or container image into the system and have it run your application. Deis and Flynn also install all of their dependencies for you through automated installers. Mesos on the other hand requires you to deploy its prerequisites on your own in order to get going, then requires you to install frameworks on top of it to make it able to schedule and run your applications.
     I wanted to make a Mesos implementation that felt as easy to make useful as Deis and Flynn. I have been working with chef-provisioning to deploy clustered applications for a while now so I figured I would use my previous techniques in order to automate the process of deploying a functional and working N node Mesos/Marathon cluster. Over the last month, I have also been able to play with Mesosphere’s DCOS so was able to get a better idea of what it takes to really make Mesos useful to end users. The “batteries included” version of Mesos is architected as follows:
container-service
Each of the machines in our Mesos cluster will run all of these components, giving us a nice symmetrical architecture for deployment. Mesos and many of its dependencies rely on a working Zookeeper as a distributed key value store. All of the state for the cluster is stored here. Luckily, for this piece of the deployment puzzle I was able to leverage the Chef community’s Exhibitor cookbook which got my ZK cluster up in a snap. Once Zookeeper was deployed, I was able to get my Mesos masters and slaves connected together and was able to see my CPU, memory and disk resources available within the Mesos cluster.
mesos-resources
    Mesos itself does not handle creating applications as services so we need to deploy a service management layer. In my case, I chose Marathon as it is intended to manage long running services like the ones I was most interested in deploying (Elasticsearch, Logstash, Kibana, Chronos). Marathon is run outside of Mesos and acts as the bootstrapper for the rest of the services that we would like to use, our distributed init system.
     Once applications are deployed into Marathon it is necessary to have a mechanism to discover where other services are running. Although it is possible to pin particular services to particular nodes through the Marathon application definition, I would prefer not to have to think about IP addressing in order to connect applications. The preferred method of service discovery in the Mesos ecosystem is to use Mesos DNS and host it as a service in Marathon across all of your nodes. Each slave node can then use itself as a DNS resolver, wherein queries for services  get handled internally and all others are recursed to an upstream DNS server.
     Now that the architecture of the container service is laid out for you, you can get to deploying your stack by heading over to the README. This deployment procedure will not only deploy Mesos+Marathon but will also deploy a full ELK into the cluster to demonstrate connecting various services together in order to provide a higher order one.
Standard
Eucalyptus, QA

EucaLoader: Load Testing Your Eucalyptus Cloud

Locust-full-page

Introduction

After provisioning a cloud that will be used by many users, it is best practice to do load or burn in testing to ensure that it meets your stability and scale requirements. These activities can be performed manually by running commands to run many  instances or create many volumes for example. In order to perform sustained long term tests it is beneficial to have an automated tool that will not only perform the test actions but also allow you to analyze and interpret the results in a simple way.

Background

Over the last year, I have been working with Locust to provide a load testing framework for Eucalyptus clouds. Locust is generally used for load testing web pages but allows for customizable clients which allowed me to hook in our Eutester library in order to generate load. Once I had created my client, I was able to create Locust “tasks” that map to activities on the cloud. Tasks are user interactions like creating a bucket or deleting a volume. Once the tasks were defined I was able to compose them into user profiles that define which types of actions each simulated user will be able to run as well as weighting their probability so that the load can most closely approximate a real world use case. In order to make the deployment of EucaLoader as simple as possible, I have baked the entire deployment into a CloudFormation template. This means that once you have the basics of your deployment done, you can start stressing your cloud and analyzing the results with minimal effort.

Using EucaLoader

Prerequisites

In order to use EucaLoader you will first need to load up an Ubuntu Trusty image into your cloud as follows:

# wget https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
# euca-install-image -i trusty-server-cloudimg-amd64-disk1.raw -n trusty -r x86_64 -b trusty --virt hvm

We will also need to clone the EucaLoader repository and install its dependencies:

# git clone https://github.com/viglesiasce/euca-loader
# pip install troposphere

Next we will upload credentials for a test account to our objectstore so that our loader can pull them down for Eutester to use:

# euare-accountcreate loader
# euca_conf --get-credentials  loader.zip --cred-account loader
# s3cmd mb s3://loader
# s3cmd put -P loader.zip s3://loader/admin.zip


Launching the stack

Once inside the euca-loader directory we will create our CloudFormation template and then create our stack by passing in the required parameters:

# ./create-locust-cfn-template.py > loader.cfn
# euform-create-stack --template-f loader.cfn  loader -p KeyName=<your-keypair-name> -p CredentialURL='http://<your-user-facing-service-ip>:8773/services/objectstorage/loader/admin.zip' -p ImageID=<emi-id-for-trusty> -p InstanceType=m1.large

At this point you should be able to monitor the stack creation with the following commands

# euform-describe-stacks
# euform-describe-stack-events loader

Once the stack shows as CREATE_COMPLETE, the describe stacks command should show outputs which point you to the Locust web portal (WebPortalUrl) and to your Grafana dashboard for monitoring trends (GrafanaURL).


Starting the tests

In order to start your user simulation, point your web browser to the the WebPortalUrl as defined by the describe stacks output. Once there you can enter the amount of users you’d like to simulate as well as how quickly those users should “hatch”.

Locust-start-test

Once you’ve started the test, the statistics for each type of requests will begin to show up in the Locust dashboard.

Locust-test-running


See your results

In order to better visualize the trends in your results, EucaLoader provides a Grafana dashboard that tracks a few of the requests for various metrics. This dashboard is easily customized to your particular test and is meant as a jumping off point.

Locust-dashboard

Standard
Uncategorized

Introducing HuevOS+RancherOS

huevosArch

Today is an exciting day in Santa Barbara. We are very pleased to introduce our latest innovation to the world of DevOps. 

HuevOS – the Docker-based open-source operating system for tomorrow’s IT and Dev/Ops professional.  HuevOS 1.0 (SunnySide) is the open-source/free-range/gluten-free solution that forms the perfect complement to RancherOS.  In addition, we’re delighted to begin development on our proprietary blend of Services and Language Software as a Service (SaLSaaS) which, when overlayed atop a HuevOS+RancherOS stack, provides a complete and delicious solution around which your whole day can be centered.

Try HuevOS+RancherOS today and let us know which SaLSaaS we should work on first to ensure your hunger for DevOps is quenched thoroughly.  

To get your first taste visit the following repository which includes our Chef Recipe and a Vagrantfile to get you HuevOS in short order:

https://github.com/viglesiasce/huevos-cookbook.git

If you already have your RancherOS host up its easy to add in our HuevOS to the mix via the Docker Registry:

docker pull viglesiasce/huevos; docker run huevos

A huge thank you to all involved in getting us to this point and being able to ship a 1.0 version of the HuevOS+RancherOS platform.

Happy clucking!!

huevOS

Standard
Eucalyptus

Deploying Cassandra and Consul with Chef Provisioning

ConsulCassandra

Introduction

Chef Provisioning (née Chef Metal) is an incredibly flexible way to deploy infrastructure. Its many plugins allow users to develop a single methodology for deploying an application that can then be repeated against many types of infrastructure (AWS, Euca, Openstack, etc). Chef provisioning is especially useful when deploying clusters of machines that make up an application as it allows for machines to be:

  • Staged before deployment
  • Batched for parallelism
  • Deployed in serial when necessary

This level of flexibility means that deploying interesting distributed systems like Cassandra and Consul is a breeze. By leveraging community cookbooks for Consul and Cassandra, we can largely ignore the details of package installation and service management and focus our time on orchestrating the stack in the correct order and configuring the necessary attributes such that our cluster converges properly. For this tutorial we will be deploying:

  • DataStax Cassandra 2.0.x
  • Consul
    • Service discovery via DNS
    • Health checks on a per node basis
  • Consul UI
    • Allows for service health visualization

Once complete we will be able to use Consul’s DNS service to load balance our Cassandra client requests across the cluster as well as use Consul UI in order to keep tabs on our clusters’ health.

In the process of writing up this methodology, I went a step further and created a repository and toolchain for configuring and managing the lifecycle of clustered deployments. The chef-provisioning-recipes repository will allow you to configure your AWS/Euca cloud credentials and images and deploy any of the clustered applications available in the repository.

Steps to reproduce

Install prerequisites

  • Install ChefDK
  • Install package deps (for CentOS 6)
    yum install python-devel gcc git
  • Install python deps:
    easy_install fabric PyYaml
  • Clone the chef-provisioning-recipes repo:
    git clone https://github.com/viglesiasce/chef-provisioning-recipes

Edit config file

The configuration file (config.yml) contains information about how and where to deploy the cluster. There are two main sections in the file:

  1. Profiles
    1. Which credentials/cloud to use
    2. What image to use
    3. What instance type to use
    4. What username to use
  2. Credentials
    1. Cloud endpoints or region
    2. Cloud access and secret keys

Edit the config.yml file found in the repo such that the default profile points to a CentOS 6 image in your cloud and the default credentials point to the proper cloud.

Run the deployment

Once the deployer has been configured we simply need to run it and tell it which cluster we would like to deploy. In this case we’d like to deploy Cassandra so we will run the deployer as follows:

./deployer.py cassandra

This will now automate the following process:

  1. Create a chef repository
  2. Download all necessary cookbooks
  3. Create all necessary instances
  4. Deploy Cassandra and Consul

Once this is complete you should be able to see your instances running in your cloud tagged as follows: cassandra-default-N. In order to access your Consul UI dashboard go to http://instance-pub-ip:8500

You should now also be able to query any of your Consul servers for the IPs of your Cassandra cluster:

nslookup cassandra.service.paas.home &amp;amp;amp;lt;instance-pub-ip&amp;amp;amp;gt;

In order to tear down the cluster simply run:

./deployer.py cassandra --op destroy
Standard