Scheduled Mutations with Kyverno
Resource mutation is a valuable ability and can be used to solve many different use cases, some of which I covered in the past here and here. The thing most mutations have in common, however, is that there needs to be some event to occur which triggers the mutation. This event is most commonly an AdmissionReview request resulting from an operation performed on a resource such as a creation, update, or deletion. There are also some cases where you want to perform a mutation on a schedule, for example to scale some Kubernetes Deployments to zero on a nightly basis or annotate Namespaces. In this short-ish post, I'll show one neat way I came up with to use Kyverno to perform mutations on a scheduled basis.
Kyverno does not have a native ability to perform a mutation on a scheduled basis. But, fortunately, there is another mechanism within Kyverno which does: cleanup policies. A cleanup policy is a Kyverno policy type which allows the user to request Kyverno perform a cleanup (i.e., deletion) of resources based upon a schedule. The schedule is given in cron format (because cleanup policies are actually powered by CronJobs), so when the scheduled time occurs, Kyverno select the resources defined in the policy and removes them.
Cleanup policies result in deletion events. And now that we have a scheduled deletion, we can harness this deletion as the input for a separate mutation rule thereby effectively mutating on a scheduled basis. The thing is we need the resource we're deleting to be available again or else the cleanup policy won't have anything to cleanup. With nothing to cleanup, there's no input trigger. What could we use? Well, how about a Pod spawned from a Deployment? When we delete a Pod owned by a controller such as a Deployment (actually, the ReplicaSet, the intermediary controller, is really the one responsible here), the desired state no longer matches the actual state and so another Pod will get brought online. That seems perfect! In total, the whole flow looks something like the following.
Now, let's put this in context with a real-world use case. We have some Deployments in a Namespace called
saturn which are of a couple varieties. There are "experimental" or ephemeral Deployments alongside some applications which need to be perpetually available, including both infrastructure-level tools and customer-facing apps. We want to scale the experimental Deployments, indicated by the label
env=dev, to zero every day at midnight but not touch any others. This may be a tiny bit contrived but it suffices to illustrate the concept.
First, we need to put some permissions in place. The manifest below includes two such permissions. The first is to allow Kyverno to delete Pods while the second allows it to mutate (i.e., update in this case) existing Deployments. Both of these are permissions Kyverno does not provide out-of-the-box.
1apiVersion: rbac.authorization.k8s.io/v1 2kind: ClusterRole 3metadata: 4 labels: 5 app.kubernetes.io/component: cleanup-controller 6 app.kubernetes.io/instance: kyverno 7 app.kubernetes.io/part-of: kyverno 8 name: kyverno:cleanup-pods 9rules: 10- apiGroups: 11 - "" 12 resources: 13 - pods 14 verbs: 15 - get 16 - list 17 - delete 18--- 19apiVersion: rbac.authorization.k8s.io/v1 20kind: ClusterRole 21metadata: 22 labels: 23 app.kubernetes.io/component: background-controller 24 app.kubernetes.io/instance: kyverno 25 app.kubernetes.io/part-of: kyverno 26 name: kyverno:update-deployments 27rules: 28- apiGroups: 29 - apps 30 resources: 31 - deployments 32 verbs: 33 - update
Next, let's simulate one of those experimental Deployments. I'll just use a simple busybox app here so you get the picture. Notice how this has the label
1apiVersion: apps/v1 2kind: Deployment 3metadata: 4 name: busybox 5 namespace: saturn 6 labels: 7 app: busybox 8 env: dev 9spec: 10 replicas: 2 11 selector: 12 matchLabels: 13 app: busybox 14 template: 15 metadata: 16 labels: 17 app: busybox 18 spec: 19 containers: 20 - image: busybox:1.35 21 name: busybox 22 command: 23 - sleep 24 - 1d
Now we get to the cleanup policy. This is a Namespaced
CleanupPolicy which lives in the
saturn Namespace and is configured to remove Pods beginning with the name
cleanmeup which also have the label
purpose=deleteme. As you can see, this is highly specific since we want to be very tactical with the Pods we clean up. I chose a Namespaced
CleanupPolicy here but you could absolutely use a cluster-scoped
ClusterCleanupPolicy instead but with an added clause which selects a specific Namespace.
1apiVersion: kyverno.io/v2alpha1 2kind: CleanupPolicy 3metadata: 4 name: clean 5 namespace: saturn 6spec: 7 match: 8 any: 9 - resources: 10 kinds: 11 - Pod 12 names: 13 - cleanmeup* 14 selector: 15 matchLabels: 16 purpose: deleteme 17 schedule: "0 0 * * *"
And now comes the Deployment supplying the Pod to be cleaned up. The purpose of this
cleanmeup Deployment is solely for Kyverno's use to function as the input schedule trigger. Once its one and only replica is removed, it'll immediately spawn another ensuring that subsequent runs of the cleanup policy have work to do.
1apiVersion: apps/v1 2kind: Deployment 3metadata: 4 name: cleanmeup 5 namespace: saturn 6 labels: 7 purpose: deleteme 8spec: 9 replicas: 1 10 selector: 11 matchLabels: 12 purpose: deleteme 13 template: 14 metadata: 15 labels: 16 purpose: deleteme 17 spec: 18 containers: 19 - image: busybox:1.35 20 name: busybox 21 command: 22 - sleep 23 - 1d
And, finally, the mutation policy for existing resources is below. As you can see from the
match block, we're very specifically matching only deletions on Pods named
cleanmeup in the
saturn Namespace. When such a deletion is observed, we want Kyverno to scale to zero any Deployments also in the
saturn Namespace (although this could be anywhere in the cluster if we omitted the
mutate.targets.namespace field) if they have the label
1apiVersion: kyverno.io/v2beta1 2kind: ClusterPolicy 3metadata: 4 name: mutate-existing 5spec: 6 rules: 7 - name: scale-dev-zero 8 match: 9 any: 10 - resources: 11 kinds: 12 - Pod 13 names: 14 - cleanmeup* 15 namespaces: 16 - saturn 17 operations: 18 - DELETE 19 mutate: 20 targets: 21 - apiVersion: apps/v1 22 kind: Deployment 23 namespace: saturn 24 patchStrategicMerge: 25 metadata: 26 labels: 27 <(env): dev 28 spec: 29 replicas: 0
With all the pieces in place, it's a simple matter of time. When the schedule occurs, Kyverno will delete the
cleanmeup Pod which will trigger the mutation policy which will scale the matching Deployments to zero. This will run on a perpetual loop thanks to the
cleanmeup Deployment respawning another Pod once the Kyverno cleanup controller deletes its managed Pod.
So, there you have it. Although it's a bit of a workaround, you can get scheduled mutations on existing resources with Kyverno. This could be useful in all sorts of ways from scaling to annotating/labeling to you name it. In a future blog, I'll show you a pretty cool use case for this ability which may really be of interest to platform teams. Until then.