Policy for Kubernetes Custom Resources
I've been hearing a couple things in the community that I wanted to take a few lines to dispel. The first is that Kyverno is fine for Kubernetes "out-of-the-box" resources like Pods and Deployments but is somehow either not capable or severely disadvantaged when it comes to working with CustomResources (CRs) defined in CRDs. The second is that because we've written our own internal CRD, we have no other option but to implement our own webhook. As you'll see in this short(ish) article, these assumptions aren't at all true and Kyverno is, in fact, no different in its treatment of these types of resources compared with Kubernetes default resources. It works the same whether they be Kubernetes default, ecosystem tooling, or even something homegrown.
First up is an ecosystem tool that is very common, cert-manager. Cert-manager is terrific so let me state that out of the gate. And cert-manager, like most, works via the CustomResourceDefinition primitive found in Kubernetes. In this regard, cert-manager is able to present resources to itself in the same way Kubernetes declares its own resources.
In this example, I want to use Kyverno to help control how the Certificate CustomResource is used to request a cert. If you aren't sure what that is, see the docs page. But let's say I, as a cluster owner and administrator, want users to be able to request their own certificates but they need guardrails. I want to ensure a couple things:
- that any
Certificatethey request only has a single DNS name. - if that DNS name is from our root site name that they use the established
ClusterIssuerand not something else.
I can do both of these very simply with a single Kyverno ClusterPolicy which has two rules based on the above.
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: cert-manager
5spec:
6 validationFailureAction: enforce
7 background: false
8 rules:
9 - name: limit-dnsnames
10 match:
11 resources:
12 kinds:
13 - Certificate
14 preconditions:
15 any:
16 - key: "{{request.object.spec.dnsNames | length(@)}}"
17 operator: GreaterThan
18 value: "1"
19 validate:
20 message: Only one dnsNames entry allowed per certificate request.
21 deny: {}
22 - name: restrict-corp-cert-issuer
23 match:
24 resources:
25 kinds:
26 - Certificate
27 validate:
28 message: When requesting a cert for this domain, you must use our corporate issuer.
29 pattern:
30 spec:
31 (dnsNames): ["*.corp.com"]
32 issuerRef:
33 name: our-corp-issuer
34 kind: ClusterIssuer
35 group: cert-manager.io
As you can tell, the kinds here under the match statement is simply Certificate. That's it. There's no difference in how that gets specified regardless of if it's a Deployment, a Certificate, or a Whatever resource. As long as it's defined in a CRD, it just gets named that simply. And if you needed to force a specific version for some reason or you had multiple resources with the same name, you could specify it as cert-manager.io/v1/certificates.
Now try the example from the certificates doc page and see what happens. Here it is below for reference:
1apiVersion: cert-manager.io/v1
2kind: Certificate
3metadata:
4 name: acme-crt
5spec:
6 secretName: acme-crt-secret
7 dnsNames:
8 - foo.example.com
9 - bar.example.com
10 issuerRef:
11 name: letsencrypt-prod
12 # We can reference ClusterIssuers by changing the kind here.
13 # The default value is Issuer (i.e. a locally namespaced Issuer)
14 kind: Issuer
15 group: cert-manager.io
Let's see what we get.
1$ kubectl apply -f certificate.yaml
2Error from server: error when creating "certificate.yaml": admission webhook "validate.kyverno.svc" denied the request:
3
4resource Certificate/default/acme-crt was blocked due to the following policies
5
6cert-manager:
7 limit-dnsnames: Only one dnsNames entry allowed per certificate request.
Kyverno read the policy, matched it to the incoming resource, and blocked it as it should. Now modify the Certificate resource until it complies. Maybe something like this.
1apiVersion: cert-manager.io/v1
2kind: Certificate
3metadata:
4 name: acme-crt
5spec:
6 secretName: acme-crt-secret
7 dnsNames:
8 - foo.corp.com
9 issuerRef:
10 name: our-corp-issuer
11 # We can reference ClusterIssuers by changing the kind here.
12 # The default value is Issuer (i.e. a locally namespaced Issuer)
13 kind: ClusterIssuer
14 group: cert-manager.io
1$ kubectl apply -f certificate.yaml
2certificate.cert-manager.io/acme-crt created
Done.
Not else much to say here, it just works as it should and is as simple as you've come to expect with Kyverno.
"Yeah, but," you might say, "cert-manager is different because it's, like, super mature and they've spent loads of time making it, IDK, 'official' or something lol."
Fair, and I can see how some might see that. So how about something you might have developed internally as your own, homegrown custom resource?
For this, I'll literally stick to the Kubernetes docs and won't lift another finger. No controller, no webhook, no operator BS, no nothing else. I'm sucking in what the docs say and blowing it out to my cluster, then seeing if Kyverno can form guardrails around it.
Let me create the CRD defined in the docs.
1apiVersion: apiextensions.k8s.io/v1
2kind: CustomResourceDefinition
3metadata:
4 # name must match the spec fields below, and be in the form: <plural>.<group>
5 name: crontabs.stable.example.com
6spec:
7 # group name to use for REST API: /apis/<group>/<version>
8 group: stable.example.com
9 # list of versions supported by this CustomResourceDefinition
10 versions:
11 - name: v1
12 # Each version can be enabled/disabled by Served flag.
13 served: true
14 # One and only one version must be marked as the storage version.
15 storage: true
16 schema:
17 openAPIV3Schema:
18 type: object
19 properties:
20 spec:
21 type: object
22 properties:
23 cronSpec:
24 type: string
25 image:
26 type: string
27 replicas:
28 type: integer
29 # either Namespaced or Cluster
30 scope: Namespaced
31 names:
32 # plural name to be used in the URL: /apis/<group>/<version>/<plural>
33 plural: crontabs
34 # singular name to be used as an alias on the CLI and for display
35 singular: crontab
36 # kind is normally the CamelCased singular type. Your resource manifests use this.
37 kind: CronTab
38 # shortNames allow shorter string to match your resource on the CLI
39 shortNames:
40 - ct
This defines a new resource called Crontab (no, it's not in any way related to a CronJob resource).
Now the Kyverno policy that restricts what the image name can be. You can see here I'm specifying the kind in GVK format as I mentioned above (just because; could have been CronTab). But it's pretty simple, right? No different from any other Kyverno policy, right?
1apiVersion: kyverno.io/v1
2kind: ClusterPolicy
3metadata:
4 name: block-crontab
5spec:
6 validationFailureAction: enforce
7 background: false
8 rules:
9 - name: block-crontab
10 match:
11 resources:
12 kinds:
13 - stable.example.com/v1/CronTab
14 validate:
15 message: Image must be "some-other-image".
16 pattern:
17 spec:
18 image: some-other-image
And now let me try to create a Crontab thingy which violates this policy (again, strictly from the docs).
1apiVersion: "stable.example.com/v1"
2kind: CronTab
3metadata:
4 name: my-new-cron-object
5spec:
6 cronSpec: "* * * * */5"
7 image: my-awesome-cron-image
1$ kubectl apply -f my-crontab.yaml
2Error from server: error when creating "my-crontab.yaml": admission webhook "validate.kyverno.svc" denied the request:
3resource CronTab/default/my-new-cron-object was blocked due to the following policies
4block-crontab:
5 block-crontab: 'validation error: Image must be "some-other-image". Rule block-crontab
6 failed at path /spec/image/'
So as you can see, and let me state this clearly as a summarization of this article: Kyverno works the same way for all Kubernetes resources regardless of their origin.