Policy for Kubernetes Custom Resources

Share on:

I've been hearing a couple things in the community that I wanted to take a few lines to dispel. The first is that Kyverno is fine for Kubernetes "out-of-the-box" resources like Pods and Deployments but is somehow either not capable or severely disadvantaged when it comes to working with CustomResources (CRs) defined in CRDs. The second is that because we've written our own internal CRD, we have no other option but to implement our own webhook. As you'll see in this short(ish) article, these assumptions aren't at all true and Kyverno is, in fact, no different in its treatment of these types of resources compared with Kubernetes default resources. It works the same whether they be Kubernetes default, ecosystem tooling, or even something homegrown.

First up is an ecosystem tool that is very common, cert-manager. Cert-manager is terrific so let me state that out of the gate. And cert-manager, like most, works via the CustomResourceDefinition primitive found in Kubernetes. In this regard, cert-manager is able to present resources to itself in the same way Kubernetes declares its own resources.

In this example, I want to use Kyverno to help control how the Certificate CustomResource is used to request a cert. If you aren't sure what that is, see the docs page. But let's say I, as a cluster owner and administrator, want users to be able to request their own certificates but they need guardrails. I want to ensure a couple things:

  1. that any Certificate they request only has a single DNS name.
  2. if that DNS name is from our root site name that they use the established ClusterIssuer and not something else.

I can do both of these very simply with a single Kyverno ClusterPolicy which has two rules based on the above.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: cert-manager
 5spec:
 6  validationFailureAction: enforce
 7  background: false
 8  rules:
 9  - name: limit-dnsnames
10    match:
11      resources:
12        kinds:
13        - Certificate
14    preconditions:
15      any:
16      - key: "{{request.object.spec.dnsNames | length(@)}}"
17        operator: GreaterThan
18        value: "1"
19    validate:
20      message: Only one dnsNames entry allowed per certificate request.
21      deny: {}
22  - name: restrict-corp-cert-issuer
23    match:
24      resources:
25        kinds:
26        - Certificate
27    validate:
28      message: When requesting a cert for this domain, you must use our corporate issuer.
29      pattern:
30        spec:
31          (dnsNames): ["*.corp.com"]
32          issuerRef:
33            name: our-corp-issuer
34            kind: ClusterIssuer
35            group: cert-manager.io

As you can tell, the kinds here under the match statement is simply Certificate. That's it. There's no difference in how that gets specified regardless of if it's a Deployment, a Certificate, or a Whatever resource. As long as it's defined in a CRD, it just gets named that simply. And if you needed to force a specific version for some reason or you had multiple resources with the same name, you could specify it as cert-manager.io/v1/certificates.

Now try the example from the certificates doc page and see what happens. Here it is below for reference:

 1apiVersion: cert-manager.io/v1
 2kind: Certificate
 3metadata:
 4  name: acme-crt
 5spec:
 6  secretName: acme-crt-secret
 7  dnsNames:
 8  - foo.example.com
 9  - bar.example.com
10  issuerRef:
11    name: letsencrypt-prod
12    # We can reference ClusterIssuers by changing the kind here.
13    # The default value is Issuer (i.e. a locally namespaced Issuer)
14    kind: Issuer
15    group: cert-manager.io

Let's see what we get.

1$ kubectl apply -f certificate.yaml 
2Error from server: error when creating "certificate.yaml": admission webhook "validate.kyverno.svc" denied the request: 
3
4resource Certificate/default/acme-crt was blocked due to the following policies
5
6cert-manager:
7  limit-dnsnames: Only one dnsNames entry allowed per certificate request.

Kyverno read the policy, matched it to the incoming resource, and blocked it as it should. Now modify the Certificate resource until it complies. Maybe something like this.

 1apiVersion: cert-manager.io/v1
 2kind: Certificate
 3metadata:
 4  name: acme-crt
 5spec:
 6  secretName: acme-crt-secret
 7  dnsNames:
 8  - foo.corp.com
 9  issuerRef:
10    name: our-corp-issuer
11    # We can reference ClusterIssuers by changing the kind here.
12    # The default value is Issuer (i.e. a locally namespaced Issuer)
13    kind: ClusterIssuer
14    group: cert-manager.io
1$ kubectl apply -f certificate.yaml 
2certificate.cert-manager.io/acme-crt created

Done.

Not else much to say here, it just works as it should and is as simple as you've come to expect with Kyverno.

"Yeah, but," you might say, "cert-manager is different because it's, like, super mature and they've spent loads of time making it, IDK, 'official' or something lol."

Fair, and I can see how some might see that. So how about something you might have developed internally as your own, homegrown custom resource?

For this, I'll literally stick to the Kubernetes docs and won't lift another finger. No controller, no webhook, no operator BS, no nothing else. I'm sucking in what the docs say and blowing it out to my cluster, then seeing if Kyverno can form guardrails around it.

Let me create the CRD defined in the docs.

 1apiVersion: apiextensions.k8s.io/v1
 2kind: CustomResourceDefinition
 3metadata:
 4  # name must match the spec fields below, and be in the form: <plural>.<group>
 5  name: crontabs.stable.example.com
 6spec:
 7  # group name to use for REST API: /apis/<group>/<version>
 8  group: stable.example.com
 9  # list of versions supported by this CustomResourceDefinition
10  versions:
11    - name: v1
12      # Each version can be enabled/disabled by Served flag.
13      served: true
14      # One and only one version must be marked as the storage version.
15      storage: true
16      schema:
17        openAPIV3Schema:
18          type: object
19          properties:
20            spec:
21              type: object
22              properties:
23                cronSpec:
24                  type: string
25                image:
26                  type: string
27                replicas:
28                  type: integer
29  # either Namespaced or Cluster
30  scope: Namespaced
31  names:
32    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
33    plural: crontabs
34    # singular name to be used as an alias on the CLI and for display
35    singular: crontab
36    # kind is normally the CamelCased singular type. Your resource manifests use this.
37    kind: CronTab
38    # shortNames allow shorter string to match your resource on the CLI
39    shortNames:
40    - ct

This defines a new resource called Crontab (no, it's not in any way related to a CronJob resource).

Now the Kyverno policy that restricts what the image name can be. You can see here I'm specifying the kind in GVK format as I mentioned above (just because; could have been CronTab). But it's pretty simple, right? No different from any other Kyverno policy, right?

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: block-crontab
 5spec:
 6  validationFailureAction: enforce
 7  background: false
 8  rules:
 9  - name: block-crontab
10    match:
11      resources:
12        kinds:
13        - stable.example.com/v1/CronTab
14    validate:
15      message: Image must be "some-other-image".
16      pattern:
17        spec:
18          image: some-other-image

And now let me try to create a Crontab thingy which violates this policy (again, strictly from the docs).

1apiVersion: "stable.example.com/v1"
2kind: CronTab
3metadata:
4  name: my-new-cron-object
5spec:
6  cronSpec: "* * * * */5"
7  image: my-awesome-cron-image
1$ kubectl apply -f my-crontab.yaml 
2Error from server: error when creating "my-crontab.yaml": admission webhook "validate.kyverno.svc" denied the request: 
3resource CronTab/default/my-new-cron-object was blocked due to the following policies
4block-crontab:
5  block-crontab: 'validation error: Image must be "some-other-image". Rule block-crontab
6    failed at path /spec/image/'

So as you can see, and let me state this clearly as a summarization of this article: Kyverno works the same way for all Kubernetes resources regardless of their origin.