Pod Security Admission versus Kyverno

Pod Security Admission is a new-ish feature in Kubernetes which provides out-of-the-box controls for the Pod Security Standards. I touch on its behavior a bit, but as it has been covered already elsewhere, in this article I really wanted to collect the pros and cons and then compare those to Kyverno.

Securing workloads, I think we all agree, is top of mind for most folks. The atomic unit of scheduling in Kubernetes is the Pod, which is responsible for running containers, which represents the workloads for which we're interested in providing security. So, therefore, we need to first and foremost look at securing Pods. In Kubernetes world, there are a variety of ways of doing this, but in this article I really wanted to get hands-on experience with the new-ish Pod Security Admission (PSA) capability and put it through its paces. Naturally, then, the next thing was how it stacks up against Kyverno. There are definitely pros and cons for each! Although there are several articles out there which go through using of the PSA controller--and I will do a bit of a recap--I want to specifically call out those pros and cons and compare those to Kyverno's pros and cons.

Pod Security Admission Overview

For those maybe not aware, as of Kubernetes v1.22, there is a new built-in admission controller called Pod Security Admission (PSA). The purpose of this controller is to effectively serve as the replacement for Pod Security Policies (PSP) but to do so using formal controls called the Pod Security Standards (PSS). The PSS are, as the name implies, just standards. They aren't a technology. Standards dictate the what should happen whereas a technology (like PSP) implements them as the how. PSA is a technology which implements the PSS, all of which is built directly into the Kubernetes API server. Starting in Kubernetes v1.23, the PSA is enabled by default. There's nothing else you need to do. This is both convenient and powerful.

In order to use PSA, one need only label a Namespace with an appropriate label. Labels have three (optionally four) parts: a prefix, a mode, and a level. The prefix is always pod-security.kubernetes.io followed by a slash. The mode dictates what the behavior should be in that Namespace the options of which are enforce (it will be blocked), audit (it will be allowed but show up in the audit log), and warn (it will be allowed but a warning will be shown to the user). The level dictates which profile of the PSS is in effect the options of which are privileged (fully unrestricted), baseline (minimally restrictive), and enforce (heavily restrictive). These levels effectively represent a number of different controls defined in the standards. For example, the baseline represents all of these checks and restricted represents all of baseline PLUS all of these. A fully-formed Namespace label which will "activate" the PSA behavior then looks something like this:

1pod-security.kubernetes.io/enforce: restricted

Once you label a Namespace, it instantly takes effect. It really is that simple, which is a great thing! With the above label, any submitted Pod which does not pass the restricted PSS profile (again, see here for what those are) will be instantly blocked. For example, for a "bad" Pod which doesn't meet the requirements to pass that profile, looks like the following:

1apiVersion: v1
2kind: Pod
3metadata:
4  name: badpod01
5spec:
6  containers:
7  - name: container01
8    image: dummyimagename

Attempt to apply this with the above-referenced label in place and you might see a message like this:

1Error from server (Forbidden): error when creating "bad.yaml":
2pods "badpod01" is forbidden:
3violates PodSecurity "restricted:latest":
4allowPrivilegeEscalation != false (container "container01" must set securityContext.allowPrivilegeEscalation=false),
5unrestricted capabilities (container "container01" must set securityContext.capabilities.drop=["ALL"]),
6runAsNonRoot != true (pod or container "container01" must set securityContext.runAsNonRoot=true),
7seccompProfile (pod or container "container01" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Additionally, you can apply multiple of these labels to the same Namespace to get a combination of behaviors. For example, if you wanted to block resources but also be notified about them, you could assign an additional label like pod-security.kubernetes.io/audit: restricted. When combined with the first, those violating Pods would be prohibited from running and also generate an event in the Kubernetes audit log.

That's about where I'll stop because there are plenty of blogs and videos which dive into the usage and capabilities more fully. Here are the ones I recommend:

• Kubernetes 1.23: Pod Security Graduates to Beta by Jim Angel and Lachlan Evenson (Blog)
• Hands on with Kubernetes Pod Security Admission by Lachlan Evenson (Blog)
• The Hitchhiker's Guide to Pod Security by Lachlan Evenson (Video)

What I want to cover from here on out are the pros and cons--some of which may not be so obvious--and how these relate to Kyverno. To be clear here, the purpose of this comparison is purely so you as a user can make the most informed decision for you.

Pod Security Admission Pros

1. Very easy to set up! It's on by default in Kubernetes v1.23+.
2. Integrated: It's built-in out-of-the-box with nothing you need to install. Just assign a label and go.
3. Embeds the PSS: No separate policies to install. It's all built-in and under the covers.
4. Can audit resources: You can allow violating resources to be created or even blocked and still know about it.
5. Can warn users: Rather than blocking, you can allow but warn your users, helpful when wanting to move them to more restrictive levels will still enforcing some basic security.
6. Version pinning: You can pin different versions of the Pod Security Standards each with a different mode. While there really haven't been any differences between versions (yet), this could be potentially useful.

Pod Security Admission Cons

1. Just Pods: As the name implies, it only works on Pods. That's a great (and necessary) thing, but there are WAY more things you need to also provide controls and guarantees for aside from just Pods.
2. Just PSS: Not as the name implies, but it only implements the Pod Security Standards. Aside from the prescribed profiles, you can neither be selective as to which controls you want to enforce, nor can you add custom controls. It's an all-or-nothing deal: either you implement ALL of the profile or you implement NONE of it.
3. No mutation: Unlike PSPs, there is no ability to change a resource. It takes action on what it sees and no more.
4. No enforcement of Pod controllers: This one is fairly major. For a profile in enforce mode, it will only block Pods emitted from a controller but NOT the actual controller itself. This results in, for example, ability to create a Deployment yet will silently block all the Pods it spawns.
5. Messages are not configurable: Whatever message it generates in any of the modes is not configurable at all. Probably not a big deal.
6. Seeing audits is painful: Being able to see audits involves digging into the API audit log. Setting up that log is a multi-step process, complex, not enabled by default, is disruptive if done retroactively, requires privileged access to the control plane, and the log cannot be viewed from inside the cluster.
7. Cluster-wide policy requires special configuration: PSA is intended to apply at the Namespace level. Getting it to apply to the entire cluster requires creation and configuration of a new AdmissionConfiguration file, which requires API and control plane access, setting a new flag, is disruptive, and is more complex.
8. Exemptions are very limited: This is one of the biggest ones. Exemptions are limited to usernames, runtimeClasses, and Namespaces. Common use cases for exemption like for Pod or container name simply aren't there let alone more complex ones like ClusterRole. And in order to get even that you still have to configure the AdmissionConfiguration file, so see the above bullet for the difficulties that imposes.
9. Can't use in a pipeline: PSA is engrained into the Kubernetes control plane, which means to test against it you have to actually submit something to the control plane. There's no standalone utility to know if, in advance, a given resource will work or not.
10. New: It's new and, as of v1.25 which will actually remove PSPs, will have only been enabled by default on three minor versions. Just like with anything else, it takes time to mature a technology, and that comes with enough people actually using it.

Let's also compare these against Kyverno's pros and cons. Note that these are only those framed against PSA and not absolutely everything.

Kyverno Pros

1. Works on anything: Kyverno works on any Kubernetes resource and not just Pods.
2. Custom policy: Kyverno will apply any policy behavior you can express, not limited to just PSS.
3. Mutation support: Kyverno has had mutation for a while now, and not just of Pods but of any resource.
4. Pre-built PSS policies: The Kyverno community has built the full suite of PSS policies here. They also closely monitor upstream guidance for any changes. In addition, they provide over 1,000 test resources (positive and negative) which can be used to verify the operability of those policies.
5. Easy cluster-wide policy: Kyverno has a specific Custom Resource called a ClusterPolicy which, as the name implies, will apply to the ENTIRE cluster. Still want PSA-like behavior? Use the Policy resource which applies to a specific Namespace. Mix and match the two. It's your choice.
6. Easy, granular exemptions: Policy in Kyverno has a dedicated exclude block which can be used to exclude a whole variety of conditions. If that's still not enough, it also has preconditions and at multiple levels. Basically, if you can express it, Kyverno can exclude or include it.
7. Automatic application to Pod controllers: A Kyverno policy written for Pods applies automatically to all known Kubernetes Pod controllers (see auto-generation here). Whatever behavior you dictate in the policy applies to all those controllers as well. There's no accepting a Deployment but silently block its violating Pods.
8. Audit results are easy: Kyverno writes audit results to a standard Kubernetes Custom Resource called a PolicyReport. Not only is this an open standard developed by the Kubernetes Policy Working Group, it doesn't involve jacking with the control plane. PolicyReports are visible in the cluster as just another resource. This means you can decouple who writes policy and who sees results. You can take that one step further and put in the all-in-one Policy Reporter tool which gives you graphical dashboards, alerts, and more.
9. CLI: Kyverno has its own CLI which allows you to test resources against a given set of policies to see how they'll react in a pipeline, long before they hit an actual cluster.
10. Broad version support: Kyverno can be installed on versions of Kubernetes all the way back to 1.16 up through the most current version, so you can get the same Pod Security Standards consistently and across versions.

Kyverno Cons

1. Add-on: Requires installation of an additional admission controller which runs in the cluster. It's very easy to set up, but nothing is as easy as being built-in to an existing system.
2. PSS is separate: Installing Kyverno doesn't by default give you PSS policies. However, the Kyverno project provides the full suite of PSS policies packaged conveniently in a Helm chart. Once Kyverno is installed, adding the PSS policies is literally as simple as a helm install command.
3. No warnings: Kyverno currently doesn't have a warn behavior. The current choices are enforce or audit, just like PSA.
4. Blocked resources aren't in audits: If a resource is blocked by Kyverno, it doesn't also appear in a Policy Report (Kyverno's equivalent of the Kubernetes audit log).
5. Version pinning requires more policies/rules: To be able to apply policy for different versions of Pod Security Standards, you'd need either additional rules or policies. Not really a big deal considering: 1) the community would provide them and 2) policy/rules are simple anyway on account of them being standard YAML. Adding a new rule is typically about 10-15 lines of ordinary YAML. Zero of that is code.

All of this said, I do like Pod Security Admission and think it's a good thing for the community, and I do recommend enabling it. Everyone should be enforcing, at the very least, the baseline profile. If you don't have any policy today and you're on one of these versions, it's really a no-brainer. But even if you're using Kyverno, OPA Gatekeeper, or something else, it's still probably nice to activate because, at the very least, it takes away some of the load from your admission controller freeing it up for things PSA can't handle today.

Saw something I missed? Drop me a line. In a future blog, I'm going to cover using both PSA and Kyverno together to see how you can extract the benefits of each one.