Policy-based Image Validation for Kubernetes with Anchore Engine

Introduction

As part of the journey to Kubernetes, you choose a set of Docker images that will house the dependencies needed to run your applications. You may also be using Docker images that were created by third parties to ship off-the-shelf software. Over the last year, the Kubernetes community has worked hard to make it possible to run almost any compute workload. You may now have operational tools such as monitoring and logging systems, as well as any number of databases and caching layers, running in your cluster alongside your application.

Each new use of Kubernetes requires that you vet the images that you will have running inside your cluster. Manual validation and introspection of images may be feasible, albeit tedious, to do for a very small number of images.  However, once you’ve outgrown that, you will need to set up automation to help ensure critical security vulnerabilities, unwanted dependency versions, and undesirable configurations don’t make their way into your clusters.

In the 1.9 release, Kubernetes added beta support for user configurable webhooks that it queries to determine whether a resource should be created. These ValidatingAdmissionWebhooks are a flexible way to dictate what configurations of resources you’d want to allow into your cluster.  To validate the security of your images, you can send each pod creation request to a webhook and return whether the image in the PodSpec adheres to your policies. For example, a naive implementation could allow pod creation only when images come from a set of whitelisted repositories and tags.

For more complicated image policies, you may need a more specialized tool that can gate your pod creation. Anchore Engine is “an open source project that provides a centralized service for inspection, analysis and certification of container images.” With its policy engine you can create and enforce rules for what you consider a production-ready image. As an example, an Anchore policy might check the following things about an image:

1. No critical package OS vulnerabilities
2. FROM always uses a tag
3. Image runs as non-root by default
4. Certain packages are not installed (ie openssh-server)

Below is a screenshot of these rules in the anchore.io policy editor:

In the next section, I’ll demonstrate how Anchore Engine can be integrated into your Kubernetes clusters to create a ValidatingWebhookConfiguration that runs all images through an Anchore policy check before allowing it into the cluster.

Mechanics of the image validation process

Infrastructure

The image validation process requires a few things to be deployed into your cluster before your policies can be enforced. Below is an architecture diagram of the components involved.

First, you’ll need to deploy Anchore Engine itself into the cluster. This can be achieved by using their Helm chart from the kubernetes/charts repository. Once installed, Anchore Engine is ready to accept requests for images to scan. Anchore doesn’t yet have a native API that Kubernetes can use to request validations. As such, we will create a small service inside of Kubernetes that can proxy requests from the ValidatingWebHookConfiguration.

The process of registering an admission server is relatively straightforward and can be accomplished with a single resource created in the Kubernetes API. Unfortunately, it takes more effort to create the server side process that will receive and respond to the admission request from Kubernetes. Thankfully, the good folks at Openshift have released a library that can be used as a base for creating admission servers. By leveraging the Generic Admission Server, we can reduce the amount of boilerplate in the server process and focus on the logic that we are interested in. In this case, we need to make a few requests to the Anchore Engine API and respond with a pass/fail result when we get back the result of Anchore’s policy check.

For instructions on setting this all up via a Helm chart, head over to the quick start instructions in the Kubernetes Anchore Image Validator repository.

Now that the server logic is handled, you can deploy it into your cluster and configure Kubernetes to start sending any requests to create Pods. With the infrastructure provisioned and ready, let’s take a look at the request path for an invocation to create a deployment in Kubernetes.

Request Flow

As an example, the following happens when a user or controller creates a Deployment:

1. The Deployment controller creates a ReplicaSet which in turn attempts to create a Pod
2. The configured ValidatingWebhook controller makes a request to the endpoint passing the PodSpec as the payload.
3. The Anchore validation controller configured earlier will take that spec, pull out the image references from each container in the Pod and then ask Anchore Engine for a policy result using the Anchore API.
4. The result is then sent back to Kubernetes.
5. If the check fails for any of the images, Kubernetes will reject the Pod creation request.

Multi Cluster Policy Management

It is common to have multiple Kubernetes clusters with varying locations, purposes, or sizes. As the number of clusters grows, it can be hard to keep configurations in sync. In the case of the image policy configuration, you can leverage Anchore’s hosted service to serve as the source of truth for all of your clusters’ image policies.

Anchore allows you to create, edit, and delete your policies in their hosted user interface, then have the policies pulled down by each of your Anchore Engine services by turning on the policy sync feature. Once policy sync is enabled, Anchore Engine will periodically check with the hosted service to see if any updates to the active policy have been made. These may include new approved images. or the addition/removal of a rule.

What’s Next?

Head over to the Kubernetes Anchore Image Validator repository to check out the code or run the quick start to provision Anchore Engine and the validating web hook in your cluster.

Big thanks to Zach Hill from Anchore for his help in getting things wired up.