2 minute read

Kyverno is a policy engine designed specifically for Kubernetes. It is based on validanting and mutating webhooks that intercept resources before they are created. This post will show how to troubleshoot Kyverno and it will dive into Kyverno resources when it seems that your policy is not applied.

Troubleshooting Kyverno

Test the policy locally

Install Kyverno CLI on your machine following one of the guides.

Prepare a policy and a resource used to test the policy, then run a dry-run test of the policy with command:

$ kyverno apply path/to/policy.yaml --resource path/to/resource.yaml

If the policy is a mutating policy, the command should output the mutated resource.

Check the installation on a local cluster

Since I was testing Kyverno installation, I used the manifest method:

$ kubectl create -f path/to/install.yaml

Create the mutating policy, then the resource and check if it is mutated:

$ kubectl create -f path/to/policy.yaml
$ kubectl create -f path/to/resource.yaml

In the logs of the Kyverno pod, you can see the log when the resource is mutated:

$ kubectl logs kyverno-67d595bb95-v6c9k -n kyverno

...
I0525 06:51:32.819131       1 mutation.go:120] webhooks/resource/mutate "msg"="mutation rules from policy applied successfully" "gvk"={"group":"apps","version":"v1","kind":"Deployment"} "kind"="Deployment" "name"="nginx" "namespace"="default" "operation"="CREATE" "policy"="inject-sidecar" "rules"=["inject-sidecar"] "uid"="2cf747fe-152c-4cf1-875a-885cca6504a3" "user"={"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]}

Check if the version of Kyverno is compatible to the version of Kubernetes

Check the version of Kubernetes server against the compatibility matrix of Kyverno:

$ kubectl version

Troubleshoot the installation on the target cluster

Install Kyverno and wait for pods to be running:

$ kubectl create -f path/to/install.yaml

Create the policy and check it is ready:

$ kubectl create -f path/to/policy.yaml
$ kubectl get clusterpolicy

Check the status of webhooks registered by Kyverno:

$ kubectl get validatingwebhookconfigurations

NAME                                                 WEBHOOKS   AGE
kyverno-cleanup-validating-webhook-cfg               1          17h
kyverno-exception-validating-webhook-cfg             1          17h
kyverno-policy-validating-webhook-cfg                1          7m45s
kyverno-resource-validating-webhook-cfg              0          7m45s

Check the mutating web hooks:

$ kubectl get mutatingwebhookconfigurations

NAME                                    WEBHOOKS   AGE
kyverno-policy-mutating-webhook-cfg     1          7m57s
kyverno-resource-mutating-webhook-cfg   0          7m57s
kyverno-verify-mutating-webhook-cfg     1          7m57s

The two webhook configurations kyverno-resource-validating-webhook-cfg and kyverno-resource-mutating-webhook-cfg are empty but as soon as you create a validating policy or a mutating policy, those webhooks are populated, respectively.

Create the policy and check again the webhook configurations:

kubectl create -f path/to/policy
kubectl get mutatingwebhookconfigurations

NAME                                    WEBHOOKS   AGE
kyverno-policy-mutating-webhook-cfg     1          7m57s
kyverno-resource-mutating-webhook-cfg   1          7m57s
kyverno-verify-mutating-webhook-cfg     1          7m57s

Understanding how Kyverno works

Webhook rule

Webhook are created dynamically based on the policies that are deployed on the cluster: if it is a mutating policy for deployments, the webhook rule is set for the kind “Deployment” of apiGroup “apps”:

$ kubectl get mutatingwebhookconfiguration kyverno-resource-mutating-webhook-cfg -o yaml
...
rules:
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - deployments
    scope: '*'
...

Webhook namespace selector

The namespace selector of the webhook defines the namespace that should not be affected by the webhook. And Kyverno defines a selector excluding the kyverno namespace to prevent applying mutating policies to the kyverno deployment itself:

$ kubectl get mutatingwebhookconfiguration kyverno-resource-mutating-webhook-cfg -o yaml
...
namespaceSelector:
    matchExpressions:
    - key: kubernetes.io/metadata.name
      operator: NotIn
      values:
      - kyverno
...

Conclusion

  • Test the policy using kyverno cli
  • Test the policy on a local cluster
  • Test the policy on the target cluster
  • Understand validating and mutating webhook configuration, rules and namespace selectors

Leave a comment

You may also enjoy

Implementing an HTTP proxy in Go

4 minute read

I was looking for a simple project to get my hands dirty with Go again and I came up with the HTTP proxy server. At a first glance, it seems a trivial project: create an HTTP server read the HTTP method, the URL, and the HOST from the request send an identical request to the target server.

Avoid name conflicts in Terraform workspaces

2 minute read

Terraform workspaces are a feature of some backends to associate many terraform states to the same configuration. It is convenient to have a workspace where you can try changes to infrastructure. When the changes have been tested, you can switch to the production workspace and apply the changes.

Change Kubernetes service type from LoadBalancer to NodePort and viceversa

2 minute read

Changing the Service type from NodePort to LoadBalancer and viceversa is useful depending on the cluster you are working on. If you are doing experiments in a local development cluster, like Kind, you don’t have a real load balancer in front of your cluster and you need to expose your services externally using a NodePort service. On contrary, if you are w...

Kubernetes ephemeral containers

1 minute read

Removing building and debugging tools from the images we use to run services on containers is one of the security best practices. However, you can’t debug a container if it is not shipped with the debugging tools. Security and easy of debug seems to mutual exclusive but Kubernetes provides a solution: ephemeralContainers. As the name suggests, they are ep...