Customizing Upstream Helm Charts with Kustomize


The Helm charts repository has seen some amazing growth over the last year. We now have over 200 applications that you can install into your Kubernetes clusters with a single command. Along with the growth in the number of applications that are maintained in the Charts repository, there has been huge growth in the number of contributions that are being received. The Charts repo gives the community a place to centralize their understanding of the best practices of how to deploy an application as well which configuration knobs should be available for those applications.

As I mentioned in my talk at Kubecon, these knobs (provided by the Helm values.yaml file) are on a spectrum. You start out on the left and inevitably work your way towards the right:


When a chart is first introduced, it is likely to have the use case of its author met. From there, other folks take the chart for a spin and figure out where it could be more flexible in the way its configured so that it can work for their use cases. With the addition of more configuration values, the chart becomes harder to test and reason about as both a user and maintainer. This tradeoff is one that is hard to manage and often ends with charts working their way towards more and more flexible chart values.

A class of changes that we have seen often in the Chart repository is customizations that enable low level changes to Kubernetes manifests to be plumbed through to the values file. Examples of this are:

By now, most of our popular charts allow this flexibility as it has been added by users who need to make these tweaks for the chart to be viable in their environments. The hallmark of these types of values is that they plumb through a K8s API specific configuration to the values.yaml that the user interacts with. At this point, advanced users are happy to be able to tweak what they need but the values.yaml file becomes longer and more confusing to reason about without in-depth Kubernetes knowledge.

There is a good portion of the PRs for upstream charts that are to add these last-mile customizations to charts. While these are very welcome and help others not have to fork or change charts downstream, there is another way to make upstream charts work better in your environment.

Kustomize is a project that came out of the CLI Special interest group. Kustomize “lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.” So given this functionality of customizing raw Kubernetes YAML, how we can we leverage it for customization of upstream Helm charts?

The answer lies in the helm template command which allows you to use Helms templating and values.yaml parameterization but instead of installing into the cluster just spits out the manifests to standard out. Kustomize can patch our chart’s manifests once they are fully rendered by Helm.



So what does this look like in practice? Lets customize the upstream Jenkins chart a bit. While this chart includes tons of ways to parameterize it using the extensive values file, I’ll try to accomplish similar things using the default configuration and Kustomize.

In this example I want to:

  1. Override the default plugins and include the Google Compute Engine plugin
  2. Increase the size of the Jenkins PVC
  3. Bump the memory and CPU made available to Jenkins (it is Java after all)
  4. Set the default password
  5. Ensure that the -Xmx and -Xms flags are passed via the JAVA_OPTS environment variable

Let’s see what the flow would look like:

  1. First, I’ll render the upstream jenkins/chart to a file called jenkins-base.yaml:

    git clone https://github.com/kubernetes/charts.git
    mkdir charts/stable/my-jenkins
    cd charts/stable
    helm template -n ci jenkins > my-jenkins/jenkins-base.yaml
  2. Now I’ll have helm template individually render out the templates I’d like to override

    helm template -n ci jenkins/ -x templates/jenkins-master-deployment.yaml > my-jenkins/master.yaml
    helm template -n ci jenkins/ -x templates/home-pvc.yaml > my-jenkins/home-pvc.yaml
    helm template -n ci jenkins/ -x templates/config.yaml > my-jenkins/config.yaml
  3. Next, I can customize each of these resources to my liking. Note here that anything that is unset in my patched manifests will take the value of the base manifests (jenkins-base.yaml) that come from the chart defaults. The end result can be found in this repo:

  4. Once we have the patches we’d like to apply we will tell Kustomize what our layout looks like via the kustomization.yaml file.
    - jenkins-base.yaml
    - config.yaml
    - home-pvc.yaml
    - master.yaml
  5. Now we can install our customized chart into our cluster.

    cd my-jenkins
    kustomize build | kubectl apply -f -

    I hit an issue here because Helm renders empty resources due to the way that charts enable/disable certain resources from being deployed. Kustomize does not handle these empty resources properly so I had to remove them manually from my jenkins-base.yaml.

  6. Now we can port forward to our Jenkins instance and login at http://localhost:8080 with username admin and password foobar:

    export JENKINS_POD=$(kubectl get pods -l app=ci-jenkins -o jsonpath='{.items[0].metadata.name}')
    kubectl port-forward $JENKINS_POD 8080


  7. Now I can check that my customizations worked. First off, I was able to log in with my custom password. YAY.
    Next, in the Installed Plugins list, I can see that the Google Compute Engine plugin was installed for me. Double YAY.gce-plugin



So this seems like a great way to customize upstream Helm charts but what am I missing out on by doing this? First, Helm is no longer controlling releases of manifests into the cluster. This means that you cannot use helm rollback or helm list or any of the helm release related commands to manage your deployments. With Helm+Kustomize model, you would do a rollback by reverting your commit and reapplying your previous manifests or by rolling forward to a configuration that works as expected and running your changes through your CD pipeline again.

Helm hooks are a bit wonky in this model since helm template doesn’t know what stage of a release you are currently on, it will dump all the hooks into your manifest. For example, in this example you’ll see that the Jenkins chart includes a test hook Pod that gets created but fails due to the deployment not being ready. Generally test hooks are run out of band of the installation.


One of the nice things about this model is that I can now tweak a chart’s implementation without needing to submit the change upstream in order to keep in sync. For example, I can add my bespoke labels and annotations to resources as I see fit. When a chart that I depend on gets updated I can simply re-compute my base using helm template and leave my patches in tact.

Another benefit of using Helm with Kustomize is that I can keep my organization-specific changes separate from the base application. This allows for developers to be able to more clearly see the per-environment changes that are going on as they have less code to parse.

One last benefit is that since release management is no longer handled by Helm, I don’t have to have a Tiller installed and I can use tooling like kubectl, Weave Flux or Spinnaker to manage my deployments.

What’s Next

I’d like to work with the Helm, Charts and Kustomize teams to make this pattern as streamlined as possible. If you have ideas on ways to make these things smoother please reach out via Twitter.


Policy-based Image Validation for Kubernetes with Anchore Engine


As part of the journey to Kubernetes, you choose a set of Docker images that will house the dependencies needed to run your applications. You may also be using Docker images that were created by third parties to ship off-the-shelf software. Over the last year, the Kubernetes community has worked hard to make it possible to run almost any compute workload. You may now have operational tools such as monitoring and logging systems, as well as any number of databases and caching layers, running in your cluster alongside your application.

Each new use of Kubernetes requires that you vet the images that you will have running inside your cluster. Manual validation and introspection of images may be feasible, albeit tedious, to do for a very small number of images.  However, once you’ve outgrown that, you will need to set up automation to help ensure critical security vulnerabilities, unwanted dependency versions, and undesirable configurations don’t make their way into your clusters.

In the 1.9 release, Kubernetes added beta support for user configurable webhooks that it queries to determine whether a resource should be created. These ValidatingAdmissionWebhooks are a flexible way to dictate what configurations of resources you’d want to allow into your cluster.  To validate the security of your images, you can send each pod creation request to a webhook and return whether the image in the PodSpec adheres to your policies. For example, a naive implementation could allow pod creation only when images come from a set of whitelisted repositories and tags.

For more complicated image policies, you may need a more specialized tool that can gate your pod creation. Anchore Engine is “an open source project that provides a centralized service for inspection, analysis and certification of container images.” With its policy engine you can create and enforce rules for what you consider a production-ready image. As an example, an Anchore policy might check the following things about an image:

  1. No critical package OS vulnerabilities
  2. FROM always uses a tag
  3. Image runs as non-root by default
  4. Certain packages are not installed (ie openssh-server)

Below is a screenshot of these rules in the anchore.io policy editor:

In the next section, I’ll demonstrate how Anchore Engine can be integrated into your Kubernetes clusters to create a ValidatingWebhookConfiguration that runs all images through an Anchore policy check before allowing it into the cluster.

Mechanics of the image validation process


The image validation process requires a few things to be deployed into your cluster before your policies can be enforced. Below is an architecture diagram of the components involved.

First, you’ll need to deploy Anchore Engine itself into the cluster. This can be achieved by using their Helm chart from the kubernetes/charts repository. Once installed, Anchore Engine is ready to accept requests for images to scan. Anchore doesn’t yet have a native API that Kubernetes can use to request validations. As such, we will create a small service inside of Kubernetes that can proxy requests from the ValidatingWebHookConfiguration.

The process of registering an admission server is relatively straightforward and can be accomplished with a single resource created in the Kubernetes API. Unfortunately, it takes more effort to create the server side process that will receive and respond to the admission request from Kubernetes. Thankfully, the good folks at Openshift have released a library that can be used as a base for creating admission servers. By leveraging the Generic Admission Server, we can reduce the amount of boilerplate in the server process and focus on the logic that we are interested in. In this case, we need to make a few requests to the Anchore Engine API and respond with a pass/fail result when we get back the result of Anchore’s policy check.

For instructions on setting this all up via a Helm chart, head over to the quick start instructions in the Kubernetes Anchore Image Validator repository.

Now that the server logic is handled, you can deploy it into your cluster and configure Kubernetes to start sending any requests to create Pods. With the infrastructure provisioned and ready, let’s take a look at the request path for an invocation to create a deployment in Kubernetes.

Request Flow

As an example, the following happens when a user or controller creates a Deployment:

  1. The Deployment controller creates a ReplicaSet which in turn attempts to create a Pod
  2. The configured ValidatingWebhook controller makes a request to the endpoint passing the PodSpec as the payload.
  3. The Anchore validation controller configured earlier will take that spec, pull out the image references from each container in the Pod and then ask Anchore Engine for a policy result using the Anchore API.
  4. The result is then sent back to Kubernetes.
  5. If the check fails for any of the images, Kubernetes will reject the Pod creation request.

Multi Cluster Policy Management

It is common to have multiple Kubernetes clusters with varying locations, purposes, or sizes. As the number of clusters grows, it can be hard to keep configurations in sync. In the case of the image policy configuration, you can leverage Anchore’s hosted service to serve as the source of truth for all of your clusters’ image policies.

Policy-based Image Validation for Kubernetes with Anchore Engine

Anchore allows you to create, edit, and delete your policies in their hosted user interface, then have the policies pulled down by each of your Anchore Engine services by turning on the policy sync feature. Once policy sync is enabled, Anchore Engine will periodically check with the hosted service to see if any updates to the active policy have been made. These may include new approved images. or the addition/removal of a rule.

What’s Next?

Head over to the Kubernetes Anchore Image Validator repository to check out the code or run the quick start to provision Anchore Engine and the validating web hook in your cluster.

Big thanks to Zach Hill from Anchore for his help in getting things wired up.


Porting my brain from Python to Go

Over the past few months, I have made it a goal to get down and dirty with Go. I have been getting more and more involved with Kubernetes lately and feel like its time that I pick up its native language. For the last 6 years, I’ve had Python as my go-to language with Ruby sneaking in when the time is right. I really enjoy Python and feel extremely productive in it. In order to get to some level of comfort with Go I knew that I would have to take a multi-faceted approach to my learning.

Obviously, the first thing I did was just get it installed on my laptop. Being a Mac and Homebrew user, I simply ran: brew install golang. So now I had the toolchain installed and could compile/run things on my laptop. Unfortunately ‘Hello World’ was not going to get me to properly port my existing programming skills into using Go as a primary/native language. I knew enough about Go to know that I need to understand how and why it was built. Additionally, I had been in interpreted language land my whole life (Perl->Python->Ruby) so was taking another leap outside of just changing dialects. At this point I took a step back from just poking at source code to properly study up on the language.

I’ve had a Safari Books Online subscription for quite some time and have leveraged it heavily when learning new technologies. This case was no different. I picked up Programming in Go: Creating Applications for the 21st Century. I found this book to be at the right level for me, existing programming experience but new to Go. I also found that there was enough context as to how to do things the Go way such that I wouldn’t just be writing a bunch of Python in a different language. After a few flights browsing the concepts, I started to read it front to back. At the same time, I found a resource specific to my task at hand, Go for Python Programmers. This was a good way to see the code I was accustomed to writing and how I might write it differently in Go. As I continued reading and studying up, I made sure to pay particular attention when looking at Go code in the wild. The ask of myself was really to understand the code and the idioms, not just glance at it as pseudocode.

There were a few things that I needed some more clarification on after reading through my study materials. I was still confused about how packaging worked in practice. For clarification, my buddy and Go expert Evan Brown pointed me to the Go Build System repository. The thing that clicked for me here was the structure of having a repository of code that is related to each other, splitting out libraries into directories, then using the cmd directory in order to make the binaries that tied things together. This repo also has a great README that shows how they have organized their code. Thanks Go Build Team!

The next thing that I needed to hash out was how exactly I would use my Object Oriented penchant in Go. For this I decided to turn to Go by Example which had a wealth of simple example code for many of the concepts I was grappling with. Things started to click for me after looking at  structsmethods, and interfaces again through a slightly different lens.

Sweet! So now I understood (more or less) how the thing worked but I hadn’t built anything with it. The next phase was figuring out how the rubber met the road and having a concrete task to accomplish with my new tool.

I didn’t have any projects off the top of my head that I could start attacking but remembered that a few moons ago I had signed up for StarFighters, a platform that promised to provide a capture the flag style game that you could code against. I looked back through their site and noticed that their first game, StockFighter, had been released. StockFighter provides a REST API that players can code against in order to manipulate a faux stock market. I didn’t know anything about the stock market but figured this would be as good a task as any to get started. I played through the first few levels by creating a few one off binaries. Then on the harder levels I started to break out my code, create libraries, workers and all kinds of other pieces software to help me complete the tasks that  StockFighter was throwing at me. One huge help in getting me comfortable with creating this larger piece of software was that I had been using Pycharm with the Go plugin. This made code navigation, refactoring, testing and execution familiar.

Shit had gotten real. I was building a thing and feeling more comfortable with each level I played.

After my foray with StockFighter, I felt like I could use a different challenge. It turns out if you start asking around if people need some software built at Google, there will be plenty of people who want to take you up on the offer. The homie Preston was working on an IoT demonstration architecture and needed a widget that could ingest messages from PubSub and then store the data in BigTable. As he explained the project, I told him that it should take me no more than 4 hours to complete the task. I hadn’t use the Go SDKs at the time so I figured that would eat up most of my time. I sat down that afternoon and started the timer. This was the litmus test. I knew it. My brain knew it. My fingers knew it.

I made it happen in just under 5 hours which to me was a damn good effort as I knew that generally estimates are off by 2x. After doing a little cross-compiling magic, I was able to ship Preston a set of binaries for Mac, Linux and Windows that achieved his task.

I’m certainly not done with my journey but I’m happy to have had small successes that make me feel at home with Go.



Building a container service with Mesos and Eucalyptus

    Over the past few months, I’ve been digging into what it means to work with a distributed container service. Inspired by Werner Vogel’s latest post about ECS, I decided to show an architecture for deploying a container service in Eucalyptus. As part of my investigations into containers, I have looked at the following platforms that provide the ability to manage container based services:
Each of these provide you with a symmetrical (all components run on all hosts) and scalable (hosts can be added after initial deployment) system for hosting your containerized workloads. They also include mechanisms for service discovery and load balancing. Deis and Flynn are both what I would call a “lightweight PaaS” akin to a private Heroku. Mesos, however, is a more flexible and open ended platform, which comes as a blessing and a curse. I was able to deploy many more applications in Mesos but it took me far longer to get a working platform up and running.
     Deis and Flynn are both “batteries included” type systems that once deployed allow you to immediately push your code or container image into the system and have it run your application. Deis and Flynn also install all of their dependencies for you through automated installers. Mesos on the other hand requires you to deploy its prerequisites on your own in order to get going, then requires you to install frameworks on top of it to make it able to schedule and run your applications.
     I wanted to make a Mesos implementation that felt as easy to make useful as Deis and Flynn. I have been working with chef-provisioning to deploy clustered applications for a while now so I figured I would use my previous techniques in order to automate the process of deploying a functional and working N node Mesos/Marathon cluster. Over the last month, I have also been able to play with Mesosphere’s DCOS so was able to get a better idea of what it takes to really make Mesos useful to end users. The “batteries included” version of Mesos is architected as follows:
Each of the machines in our Mesos cluster will run all of these components, giving us a nice symmetrical architecture for deployment. Mesos and many of its dependencies rely on a working Zookeeper as a distributed key value store. All of the state for the cluster is stored here. Luckily, for this piece of the deployment puzzle I was able to leverage the Chef community’s Exhibitor cookbook which got my ZK cluster up in a snap. Once Zookeeper was deployed, I was able to get my Mesos masters and slaves connected together and was able to see my CPU, memory and disk resources available within the Mesos cluster.
    Mesos itself does not handle creating applications as services so we need to deploy a service management layer. In my case, I chose Marathon as it is intended to manage long running services like the ones I was most interested in deploying (Elasticsearch, Logstash, Kibana, Chronos). Marathon is run outside of Mesos and acts as the bootstrapper for the rest of the services that we would like to use, our distributed init system.
     Once applications are deployed into Marathon it is necessary to have a mechanism to discover where other services are running. Although it is possible to pin particular services to particular nodes through the Marathon application definition, I would prefer not to have to think about IP addressing in order to connect applications. The preferred method of service discovery in the Mesos ecosystem is to use Mesos DNS and host it as a service in Marathon across all of your nodes. Each slave node can then use itself as a DNS resolver, wherein queries for services  get handled internally and all others are recursed to an upstream DNS server.
     Now that the architecture of the container service is laid out for you, you can get to deploying your stack by heading over to the README. This deployment procedure will not only deploy Mesos+Marathon but will also deploy a full ELK into the cluster to demonstrate connecting various services together in order to provide a higher order one.
Eucalyptus, QA

EucaLoader: Load Testing Your Eucalyptus Cloud



After provisioning a cloud that will be used by many users, it is best practice to do load or burn in testing to ensure that it meets your stability and scale requirements. These activities can be performed manually by running commands to run many  instances or create many volumes for example. In order to perform sustained long term tests it is beneficial to have an automated tool that will not only perform the test actions but also allow you to analyze and interpret the results in a simple way.


Over the last year, I have been working with Locust to provide a load testing framework for Eucalyptus clouds. Locust is generally used for load testing web pages but allows for customizable clients which allowed me to hook in our Eutester library in order to generate load. Once I had created my client, I was able to create Locust “tasks” that map to activities on the cloud. Tasks are user interactions like creating a bucket or deleting a volume. Once the tasks were defined I was able to compose them into user profiles that define which types of actions each simulated user will be able to run as well as weighting their probability so that the load can most closely approximate a real world use case. In order to make the deployment of EucaLoader as simple as possible, I have baked the entire deployment into a CloudFormation template. This means that once you have the basics of your deployment done, you can start stressing your cloud and analyzing the results with minimal effort.

Using EucaLoader


In order to use EucaLoader you will first need to load up an Ubuntu Trusty image into your cloud as follows:

# wget https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
# euca-install-image -i trusty-server-cloudimg-amd64-disk1.raw -n trusty -r x86_64 -b trusty --virt hvm

We will also need to clone the EucaLoader repository and install its dependencies:

# git clone https://github.com/viglesiasce/euca-loader
# pip install troposphere

Next we will upload credentials for a test account to our objectstore so that our loader can pull them down for Eutester to use:

# euare-accountcreate loader
# euca_conf --get-credentials  loader.zip --cred-account loader
# s3cmd mb s3://loader
# s3cmd put -P loader.zip s3://loader/admin.zip

Launching the stack

Once inside the euca-loader directory we will create our CloudFormation template and then create our stack by passing in the required parameters:

# ./create-locust-cfn-template.py > loader.cfn
# euform-create-stack --template-f loader.cfn  loader -p KeyName=<your-keypair-name> -p CredentialURL='http://<your-user-facing-service-ip>:8773/services/objectstorage/loader/admin.zip' -p ImageID=<emi-id-for-trusty> -p InstanceType=m1.large

At this point you should be able to monitor the stack creation with the following commands

# euform-describe-stacks
# euform-describe-stack-events loader

Once the stack shows as CREATE_COMPLETE, the describe stacks command should show outputs which point you to the Locust web portal (WebPortalUrl) and to your Grafana dashboard for monitoring trends (GrafanaURL).

Starting the tests

In order to start your user simulation, point your web browser to the the WebPortalUrl as defined by the describe stacks output. Once there you can enter the amount of users you’d like to simulate as well as how quickly those users should “hatch”.


Once you’ve started the test, the statistics for each type of requests will begin to show up in the Locust dashboard.


See your results

In order to better visualize the trends in your results, EucaLoader provides a Grafana dashboard that tracks a few of the requests for various metrics. This dashboard is easily customized to your particular test and is meant as a jumping off point.



Introducing HuevOS+RancherOS


Today is an exciting day in Santa Barbara. We are very pleased to introduce our latest innovation to the world of DevOps. 

HuevOS – the Docker-based open-source operating system for tomorrow’s IT and Dev/Ops professional.  HuevOS 1.0 (SunnySide) is the open-source/free-range/gluten-free solution that forms the perfect complement to RancherOS.  In addition, we’re delighted to begin development on our proprietary blend of Services and Language Software as a Service (SaLSaaS) which, when overlayed atop a HuevOS+RancherOS stack, provides a complete and delicious solution around which your whole day can be centered.

Try HuevOS+RancherOS today and let us know which SaLSaaS we should work on first to ensure your hunger for DevOps is quenched thoroughly.  

To get your first taste visit the following repository which includes our Chef Recipe and a Vagrantfile to get you HuevOS in short order:


If you already have your RancherOS host up its easy to add in our HuevOS to the mix via the Docker Registry:

docker pull viglesiasce/huevos; docker run huevos

A huge thank you to all involved in getting us to this point and being able to ship a 1.0 version of the HuevOS+RancherOS platform.

Happy clucking!!