Kubernetes Policy Comparison: OPA/Gatekeeper vs Kyverno
(Last Updated November 2023)
Although Pod Security Policies (PSPs) are a distant memory for few having been removed in Kubernetes 1.25, many are still tied to versions where they are in use with no clear plan for how they will migrate. An alternative solution in the form of Pod Security Admission (PSA) has been provided, however it operates fundamentally differently than PSP and has many gaps which some organizations find unsuitable. Outside of this, the two leading CNCF projects which are prime candidates for PSP/PSA replacement are Open Policy Agent (OPA) via Gatekeeper and Kyverno, each with their own strengths and weaknesses. But it is confusing and daunting to compare and contrast them given that no comparisons presently exist. Not only are these projects viable replacements for PSP/PSA, they are vastly more capable than simply acting on Pods alone--or even performing validation alone--they are full Kubernetes policy engines. The goal of this article is to provide a neutral and impartial comparison between Gatekeeper and Kyverno so users may make the most informed selection decision for themselves. It will be an evaluation of each project's merits assuming only the Kubernetes angle.
Because this article is being written with Kubernetes in mind, the OPA/Gatekeeper project will simply be referred to as Gatekeeper henceforth.
This comparison article is organized into two main sections bookended by an introduction and closing. The first is an objective comparison, in tabular format, of Gatekeeper and Kyverno focusing on three main categories of attributes and capabilities reflecting, as best as possible, only the facts. The second is my subjective analysis given the earlier comparison and will provide personal opinion and commentary as to use case and recommendations of the ideal user.
I'd like to openly state my own personal bias in the interest of transparency. I am a Kyverno contributor and not a Gatekeeper one. I have authored several blogs on Kyverno at this point and none on Gatekeeper. I have also been somewhat critical of OPA/Rego in the past. However, my goal here is to put all that plus any personal feelings aside and attempt to approach both projects fresh, with no bias and no preference for one over another.
This article has been written by equally engaging both the Kyverno and Open Policy Agent communities, giving the maintainers and contributors a fair and equal opportunity to comment on the comparison criteria as well as the final results of this article. No preference has been given to either project in terms of engagement or consideration either in regards to the time to comment or the methods in which to engage.
Introduction
What is a Kubernetes Policy
While Pod Security Policy in Kubernetes is a set of mechanisms for ensuring validating controls over Pods and their attributes, as the name would imply, it only operates on Pods and nothing else. Further, PSP is almost always used to block their creation and performs very limited remediation. Contrast that with policy engines such as Gatekeeper and Kyverno, and the capabilities are far more broad (i.e., applicable to more than just Pods) and deep (i.e., more than just simple validation). Having a policy engine for Kubernetes can be thought of as a way to more holistically control the Kubernetes environment and not just a single domain.
Gatekeeper Introduction
Gatekeeper is an open-source project and collaboration between a number of companies including Google and Microsoft, later donated to the CNCF. Now in its third iteration, Gatekeeper is the Kubernetes-specific implementation of Open Policy Agent (OPA), a general-purpose policy engine. Because of the relationship between Open Policy Agent with Gatekeeper, the project is often written "OPA/Gatekeeper" to acknowledge these ties. Gatekeeper enables request validation and, most recently, mutation. A major defining characteristic of Open Policy Agent is its reliance upon the use of a specialized programming language called Rego in order to implement the logic necessary for policy decisions. Through Rego, OPA is designed to be broadly applicable to disparate software and not Kubernetes exclusively, as well as achieve a high level of logic execution.
Kyverno Introduction
Kyverno is an open-source project originally from Nirmata and later donated to the CNCF where it is currently an Incubating project. Like Gatekeeper, Kyverno is a Kubernetes policy engine with validation and mutation abilities, however it adds other capabilities like image verification, resource generation, and cleanup capabilities. Unlike Gatekeeper, Kyverno was written from the start solely for use in Kubernetes and later added general-purpose JSON validation abilities similar to OPA. A major defining characteristic of Kyverno is its generation ability and the architecture which does not require specialized knowledge of a programming language to render policy decisions. Primarily as a reaction to this requirement of a specialized language, Kyverno differentiates itself from Gatekeeper through its model of simplicity.
Comparison
The following are a set of three tables which attempt to categorize and enumerate the features and qualities of each project in the most objective manner possible. The categories are features/capabilities which describe technical attributes; community/ecosystem which describe adoption and organizational attributes; and meta/misc which describe cognitive or miscellaneous attributes.
| Features/Capabilities | Gatekeeper | Kyverno |
|---|---|---|
| Validation | ✓ | ✓ |
| Mutation | ✓ᵃ | ✓ |
| Mutation for existing resources (retroactive) | X | ✓ |
| Generation | X | ✓ |
| Cleanup | X | ✓ᵃ |
| Image Verification | ✓ (via extensions) | ✓ᵃ |
| Manifest Verification (Sigstore) | X | ✓ |
| Image Registry lookups | ✓ (via extensions) | ✓ |
| Extensions | ✓ | ✓ |
| External data | ✓ | ✓ |
| Resource caching | ✓ | Xᵈ |
| Policy as native resources | ✓ | ✓ |
| Policy for custom resources | ✓ | ✓ |
| Metrics exposed | ✓ | ✓ |
OpenAPI validation schema (kubectl explain) |
Xᵇ | ✓ |
| High Availability | ✓ | ✓ |
| API object lookup | ✓ | ✓ |
| CLI with test ability | ✓ᶜ | ✓ |
| Policy audit ability | ✓ | ✓ |
| Self-service reports | X | ✓ |
| Self-service exceptions | X | ✓ᵃ |
| ValidatingAdmissionPolicy integration | X | ✓ᵃ |
ᵃ Beta status
ᵇ Not available with all CRDs
ᶜ Separate CLI
ᵈ ConfigMaps only
| Community/Ecosystem | Gatekeeper | Kyverno |
|---|---|---|
| CNCF status | Graduated (OPA) | Incubation |
| Partner ecosystem adoption* | ◗ | ◔ |
| GitHub status (stars, forks, releases, commits) | 3,310, 707, 89, 1,448 | 4,638, 693, 194, 6,767 |
| Slack channel members | 2,332 | 2,950ᵃ |
| Community traction** | ◗ | ◕ |
| Policy sample library | ✓ | ✓ |
| Number of community policy samples | 50 | 303 |
* Not well defined. Gatekeeper clearly has more adoption from partners than Kyverno, but total penetration unknown.
** No objective measurement exists. Seems obvious that Kyverno has more traction at this point in time.
ᵃ Total from two channels in two different workspaces (CNCF and Kubernetes)
| Meta/Misc | Gatekeeper | Kyverno |
|---|---|---|
| Programming required | ✓ | X |
| Use outside Kubernetes | ✓ | ✓ |
| Birth (Age as of November 2023) | July 2017 (6 years, 4 months) | May 2019 (4 years, 6 month) |
| Origin company | Styra (OPA) | Nirmata |
| Documentation maturity | ◗* | ◕ |
* Not totally objective with direct comparison being difficult. Assessment made based on Gatekeeper project/functionality and not maturity level of Rego enablement materials/literature.
Analysis
Based upon the earlier table of feature comparisons between Gatekeeper and Kyverno, I have put together a concise list of the main advantages and disadvantages of each project. This is not an exhaustive list, only the large topics.
Gatekeeper Advantages
- Capable of expressing very complex policy
- Language unification with OPA
Gatekeeper Disadvantages
- Requiring a programming language is a serious learning curve which generates large amounts of technical debt and increases time to value
- Mutation ability is in nascent stage
- No generation or other abilities means its usefulness is limited to predominantly validation use cases
- Policy is complex, often lengthy, and requires multiple documents to implement
Kyverno Advantages
- Dramatic simplicity of policy expression allowing Kubernetes-style composition
- Solid mutation abilities
- Built-in image verification reduces external complexities
- Generation, syncing, cleanup, and other abilities are unique and unmatched by any other policy engine
- Quick time to value and high degree of flexibility
Kyverno Disadvantages
- Extremely complex policy may not be possible since no programming language is exposed, or may require deferring to an external system for processing
Warning: The following is my opinionated analysis synthesized of the previous comparison table and advantages/disadvantages list in addition to my own experiences with both tools and time spent in the cloud-native community. If you are not interested in reading my opinion, this is the end of the article for you.
Kubernetes is a declarative system in which the user's desires are stated and Kubernetes, through its various controllers, goes to work reconciling the observed state in order to bring it into alignment with the desired state. This is the core value proposition of a cloud-native platform. In order to achieve this, the onus of the logic is shifted away from the user and into the platform itself. For each resource type, there exists some internal logic which houses the intelligence needed to remediate its state. With Gatekeeper, by far the largest weakness is its requirement of a purpose-built programming language called Rego, which aside from OPA, can be used nowhere else, in order to implement this logic. This is a reality because OPA is a general purpose policy engine. Only through Gatekeeper is its ability harnessed by adapting it into a Kubernetes form factor. Effectively then, the user is responsible for describing the object (policy) they wish reconciled as well as providing the logic necessary (Rego) to reconcile it. Using an external DSL to manage Kubernetes policies can become cumbersome and complex in a number of respects and add technical debt to projects. As a tradeoff, the distinct advantage is that it enables very powerful policies. After all, when one is required to write a programming language, one is only limited by the capabilities of that language and its inputs. This cost can be reduced if OPA can be leveraged elsewhere, however.
Kyverno is seen as a direct reaction to the complex technical demands required by Gatekeeper. Because it was built specifically for Kubernetes and expresses policy declaratively, its mental model is identical to the way Kubernetes objects are described and reconciled. The logic needed to carry out the policy decision is removed from the burden of the user and becomes the domain of the tool itself. This model results in an overwhelming simplification of how policy is written, which in turn makes policy easier in a number of respects. By having mutation and generation abilities, Kyverno is transfigured away from a simple admission controller and into the form of a true automation tool. By combining these many abilities, Kyverno is able to perform tasks that Gatekeeper simply cannot, but in addition has the power to eliminate other and disparate tools that might be employed in scattershot fashion throughout the cluster and/or organization. This simplicity coupled with its automation abilities and consolidation of other tools delivers tremendous value to new as well as experienced users and operators.
Based upon the information presented, it is my view that Kyverno is the best choice for application of Kubernetes policy unless a user fits into either or both of two broad technical use cases, and then the selection of Gatekeeper is warranted.
- A significant investment in OPA has already been made.
Counter: Policies written in Rego for use by OPA cannot be "recycled" by Gatekeeper; policies must be written anew. Since the learning curve and complexity of authoring and troubleshooting Rego is significant, even an existing investment in OPA will still require a from-scratch approach when implementing policy for Kubernetes. While the advantage may still be tool (or at least language) unification, the cost is still quite high. An organization using OPA can still save by implementing Kyverno in their Kubernetes clusters because of the simple policy language, plethora of sample policies, and vibrant community.
TL;DR: If your organization is already using OPA today and has made a sizeable investment, it may make sense to adopt Gatekeeper because of its similar reliance on Rego.
- The types of policies required are extremely complex in nature.
Counter: In my experience, most Kubernetes organizations/users today are using no policy whatsoever, including PSPs. And an AWS 2020 survey of customers running containerized workloads on AWS found similar results with only 49% using policy. Of the ones who are, the vast majority are expressing the same things repeatedly. Examples of this repetition, in plain language, include desires such as "containers should not be privileged" or "ensure all Namespaces carry a given label" or "validate Pods are not using hostPath volumes". While the term "complex" is relative and somewhat subjective, such policy expressions are decidedly not complex. Kyverno allows policy to be written in the simplest form possible which in turn is easier to reason about and maintain. Paying an additional price for a tool which is more complex and difficult yields no benefit if it isn't being used to its fullest potential.
TL;DR: If you are not expressing extremely complex policy, Gatekeeper gives you no advantage here.
Closing
Both the Gatekeeper and Kyverno projects are valuable and capable policy engines in their own right, each with strengths and weaknesses. Ultimately, users should evaluate and make the most informed decision given their use cases and constraints, but as a general recommendation, all production users should plan on using a policy engine to secure their clusters and simplify Kubernetes management.