Customizing Upstream Helm Charts with Kustomize


The Helm charts repository has seen some amazing growth over the last year. We now have over 200 applications that you can install into your Kubernetes clusters with a single command. Along with the growth in the number of applications that are maintained in the Charts repository, there has been huge growth in the number of contributions that are being received. The Charts repo gives the community a place to centralize their understanding of the best practices of how to deploy an application as well which configuration knobs should be available for those applications.

As I mentioned in my talk at Kubecon, these knobs (provided by the Helm values.yaml file) are on a spectrum. You start out on the left and inevitably work your way towards the right:


When a chart is first introduced, it is likely to have the use case of its author met. From there, other folks take the chart for a spin and figure out where it could be more flexible in the way its configured so that it can work for their use cases. With the addition of more configuration values, the chart becomes harder to test and reason about as both a user and maintainer. This tradeoff is one that is hard to manage and often ends with charts working their way towards more and more flexible chart values.

A class of changes that we have seen often in the Chart repository is customizations that enable low level changes to Kubernetes manifests to be plumbed through to the values file. Examples of this are:

By now, most of our popular charts allow this flexibility as it has been added by users who need to make these tweaks for the chart to be viable in their environments. The hallmark of these types of values is that they plumb through a K8s API specific configuration to the values.yaml that the user interacts with. At this point, advanced users are happy to be able to tweak what they need but the values.yaml file becomes longer and more confusing to reason about without in-depth Kubernetes knowledge.

There is a good portion of the PRs for upstream charts that are to add these last-mile customizations to charts. While these are very welcome and help others not have to fork or change charts downstream, there is another way to make upstream charts work better in your environment.

Kustomize is a project that came out of the CLI Special interest group. Kustomize “lets you customize raw, template-free YAML files for multiple purposes, leaving the original YAML untouched and usable as is.” So given this functionality of customizing raw Kubernetes YAML, how we can we leverage it for customization of upstream Helm charts?

The answer lies in the helm template command which allows you to use Helms templating and values.yaml parameterization but instead of installing into the cluster just spits out the manifests to standard out. Kustomize can patch our chart’s manifests once they are fully rendered by Helm.



So what does this look like in practice? Lets customize the upstream Jenkins chart a bit. While this chart includes tons of ways to parameterize it using the extensive values file, I’ll try to accomplish similar things using the default configuration and Kustomize.

In this example I want to:

  1. Override the default plugins and include the Google Compute Engine plugin
  2. Increase the size of the Jenkins PVC
  3. Bump the memory and CPU made available to Jenkins (it is Java after all)
  4. Set the default password
  5. Ensure that the -Xmx and -Xms flags are passed via the JAVA_OPTS environment variable

Let’s see what the flow would look like:

  1. First, I’ll render the upstream jenkins/chart to a file called jenkins-base.yaml:

    git clone https://github.com/kubernetes/charts.git
    mkdir charts/stable/my-jenkins
    cd charts/stable
    helm template -n ci jenkins > my-jenkins/jenkins-base.yaml
  2. Now I’ll have helm template individually render out the templates I’d like to override

    helm template -n ci jenkins/ -x templates/jenkins-master-deployment.yaml > my-jenkins/master.yaml
    helm template -n ci jenkins/ -x templates/home-pvc.yaml > my-jenkins/home-pvc.yaml
    helm template -n ci jenkins/ -x templates/config.yaml > my-jenkins/config.yaml
  3. Next, I can customize each of these resources to my liking. Note here that anything that is unset in my patched manifests will take the value of the base manifests (jenkins-base.yaml) that come from the chart defaults. The end result can be found in this repo:

  4. Once we have the patches we’d like to apply we will tell Kustomize what our layout looks like via the kustomization.yaml file.
    - jenkins-base.yaml
    - config.yaml
    - home-pvc.yaml
    - master.yaml
  5. Now we can install our customized chart into our cluster.

    cd my-jenkins
    kustomize build | kubectl apply -f -

    I hit an issue here because Helm renders empty resources due to the way that charts enable/disable certain resources from being deployed. Kustomize does not handle these empty resources properly so I had to remove them manually from my jenkins-base.yaml.

  6. Now we can port forward to our Jenkins instance and login at http://localhost:8080 with username admin and password foobar:

    export JENKINS_POD=$(kubectl get pods -l app=ci-jenkins -o jsonpath='{.items[0].metadata.name}')
    kubectl port-forward $JENKINS_POD 8080


  7. Now I can check that my customizations worked. First off, I was able to log in with my custom password. YAY.
    Next, in the Installed Plugins list, I can see that the Google Compute Engine plugin was installed for me. Double YAY.gce-plugin



So this seems like a great way to customize upstream Helm charts but what am I missing out on by doing this? First, Helm is no longer controlling releases of manifests into the cluster. This means that you cannot use helm rollback or helm list or any of the helm release related commands to manage your deployments. With Helm+Kustomize model, you would do a rollback by reverting your commit and reapplying your previous manifests or by rolling forward to a configuration that works as expected and running your changes through your CD pipeline again.

Helm hooks are a bit wonky in this model since helm template doesn’t know what stage of a release you are currently on, it will dump all the hooks into your manifest. For example, in this example you’ll see that the Jenkins chart includes a test hook Pod that gets created but fails due to the deployment not being ready. Generally test hooks are run out of band of the installation.


One of the nice things about this model is that I can now tweak a chart’s implementation without needing to submit the change upstream in order to keep in sync. For example, I can add my bespoke labels and annotations to resources as I see fit. When a chart that I depend on gets updated I can simply re-compute my base using helm template and leave my patches in tact.

Another benefit of using Helm with Kustomize is that I can keep my organization-specific changes separate from the base application. This allows for developers to be able to more clearly see the per-environment changes that are going on as they have less code to parse.

One last benefit is that since release management is no longer handled by Helm, I don’t have to have a Tiller installed and I can use tooling like kubectl, Weave Flux or Spinnaker to manage my deployments.

What’s Next

I’d like to work with the Helm, Charts and Kustomize teams to make this pattern as streamlined as possible. If you have ideas on ways to make these things smoother please reach out via Twitter.


Policy-based Image Validation for Kubernetes with Anchore Engine


As part of the journey to Kubernetes, you choose a set of Docker images that will house the dependencies needed to run your applications. You may also be using Docker images that were created by third parties to ship off-the-shelf software. Over the last year, the Kubernetes community has worked hard to make it possible to run almost any compute workload. You may now have operational tools such as monitoring and logging systems, as well as any number of databases and caching layers, running in your cluster alongside your application.

Each new use of Kubernetes requires that you vet the images that you will have running inside your cluster. Manual validation and introspection of images may be feasible, albeit tedious, to do for a very small number of images.  However, once you’ve outgrown that, you will need to set up automation to help ensure critical security vulnerabilities, unwanted dependency versions, and undesirable configurations don’t make their way into your clusters.

In the 1.9 release, Kubernetes added beta support for user configurable webhooks that it queries to determine whether a resource should be created. These ValidatingAdmissionWebhooks are a flexible way to dictate what configurations of resources you’d want to allow into your cluster.  To validate the security of your images, you can send each pod creation request to a webhook and return whether the image in the PodSpec adheres to your policies. For example, a naive implementation could allow pod creation only when images come from a set of whitelisted repositories and tags.

For more complicated image policies, you may need a more specialized tool that can gate your pod creation. Anchore Engine is “an open source project that provides a centralized service for inspection, analysis and certification of container images.” With its policy engine you can create and enforce rules for what you consider a production-ready image. As an example, an Anchore policy might check the following things about an image:

  1. No critical package OS vulnerabilities
  2. FROM always uses a tag
  3. Image runs as non-root by default
  4. Certain packages are not installed (ie openssh-server)

Below is a screenshot of these rules in the anchore.io policy editor:

In the next section, I’ll demonstrate how Anchore Engine can be integrated into your Kubernetes clusters to create a ValidatingWebhookConfiguration that runs all images through an Anchore policy check before allowing it into the cluster.

Mechanics of the image validation process


The image validation process requires a few things to be deployed into your cluster before your policies can be enforced. Below is an architecture diagram of the components involved.

First, you’ll need to deploy Anchore Engine itself into the cluster. This can be achieved by using their Helm chart from the kubernetes/charts repository. Once installed, Anchore Engine is ready to accept requests for images to scan. Anchore doesn’t yet have a native API that Kubernetes can use to request validations. As such, we will create a small service inside of Kubernetes that can proxy requests from the ValidatingWebHookConfiguration.

The process of registering an admission server is relatively straightforward and can be accomplished with a single resource created in the Kubernetes API. Unfortunately, it takes more effort to create the server side process that will receive and respond to the admission request from Kubernetes. Thankfully, the good folks at Openshift have released a library that can be used as a base for creating admission servers. By leveraging the Generic Admission Server, we can reduce the amount of boilerplate in the server process and focus on the logic that we are interested in. In this case, we need to make a few requests to the Anchore Engine API and respond with a pass/fail result when we get back the result of Anchore’s policy check.

For instructions on setting this all up via a Helm chart, head over to the quick start instructions in the Kubernetes Anchore Image Validator repository.

Now that the server logic is handled, you can deploy it into your cluster and configure Kubernetes to start sending any requests to create Pods. With the infrastructure provisioned and ready, let’s take a look at the request path for an invocation to create a deployment in Kubernetes.

Request Flow

As an example, the following happens when a user or controller creates a Deployment:

  1. The Deployment controller creates a ReplicaSet which in turn attempts to create a Pod
  2. The configured ValidatingWebhook controller makes a request to the endpoint passing the PodSpec as the payload.
  3. The Anchore validation controller configured earlier will take that spec, pull out the image references from each container in the Pod and then ask Anchore Engine for a policy result using the Anchore API.
  4. The result is then sent back to Kubernetes.
  5. If the check fails for any of the images, Kubernetes will reject the Pod creation request.

Multi Cluster Policy Management

It is common to have multiple Kubernetes clusters with varying locations, purposes, or sizes. As the number of clusters grows, it can be hard to keep configurations in sync. In the case of the image policy configuration, you can leverage Anchore’s hosted service to serve as the source of truth for all of your clusters’ image policies.

Policy-based Image Validation for Kubernetes with Anchore Engine

Anchore allows you to create, edit, and delete your policies in their hosted user interface, then have the policies pulled down by each of your Anchore Engine services by turning on the policy sync feature. Once policy sync is enabled, Anchore Engine will periodically check with the hosted service to see if any updates to the active policy have been made. These may include new approved images. or the addition/removal of a rule.

What’s Next?

Head over to the Kubernetes Anchore Image Validator repository to check out the code or run the quick start to provision Anchore Engine and the validating web hook in your cluster.

Big thanks to Zach Hill from Anchore for his help in getting things wired up.