Restart Kubernetes pods following a schedule using Kubectl

Many times we need to restart pods daily or after some hours for diverse reasons, for example in the case of API projects with memory leaks, in that case the allocated memory is not released after its use, leading to increased memory consumption over time, causing degraded performance and potential crashes, in addition, in a Kubernetes environment that can throw OOM (Out-of-Memory) errors or saturate the Kubernetes node resources.

That’s why restarting those pods on a schedule can help solve those problems, at least until the cause of the problem can be resolved.

The easiest way to do this in Kubernetes is to deploy your project using ReplicaSet or any workload that internally uses it (like a Kubernetes Deployment) because when you delete those pods, ReplicaSet will automatically create new ones when it detects they are no longer running.

Practical case

Let’s imagine that we want to restart all the pods of this Kubernetes Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
  namespace: apps
spec:
  replicas: 5
  selector:
    matchLabels:
      app: app1
  template:
    metadata:
      labels:
        app: app1
    spec:
      containers:
        - image: crccheck/hello-world
          name: hello-world

As you can see, it uses the label “app”, so, to delete all the pods with the same label, we can use this Kubectl command:

kubectl delete -n apps pods --selector=app="app1"

So, we need to run that Kubectl command on a schedule. For that, we will use a Kubernetes CronJob.

A Kubernetes CronJob creates a Kubernetes Job on a repeating schedule.

The Kubernetes CronJob container will execute Kubectl commands, so the container image must be able to run them.

We can use any available container image that supports it in the Docker Hub or even create it ourselves.

For this guide, I will use bitnami/kubectl:1.31.3.

Now the Kubernetes CronJob container can run Kubectl commands, but this is not enough, because it doesn’t have permission to use any Kubectl command.

First, we need to give an identity to the pod and this identity must have enough permissions to use the kubectl delete pods command.

In that order of ideas, we need to:

  1. Create an identity for the pod for which we will use a Kubernetes ServiceAccount.
  2. Create a set of permissions for that identity for which we will use a Kubernetes Role.
  3. Assign the set of permissions to the identity, for which we will use a Kubernetes RoleBinding.
  4. Deploy a Kubernetes CronJob with the identity previously mentioned.

Process

In this post we will use only a bash terminal, deploying all using Kubectl commands.

Define variables

First, we need to define the following variables in our bash terminal:

# The name of the app from which we will delete its pods.
AppName="app1"

# The namespace of the app from which we will delete its pods.
AppNamespace="apps"

# The name that the ServiceAccount will have.
ServiceAccountName="${AppName}-pod-deleter"

# The name that the Role will have.
RoleName="delete-pods"

Create the ServiceAccount

kubectl create -n ${AppNamespace} serviceaccount ${ServiceAccountName}

Create the Role

  • The target resource is pods.
  • The necessary permissions/verbs are: list and delete.
kubectl create role ${RoleName} --namespace ${AppNamespace} --resource=pods --verb=list,delete

Create the RoleBinding

kubectl create rolebinding ${RoleName}-${ServiceAccountName}-binding --namespace ${AppNamespace} --role=${RoleName} --serviceaccount=${AppNamespace}:${ServiceAccountName}

Verify ServiceAccount permissions

Before creating the CronJob we can check if the ServiceAccount has permission to use the verb::

# Should return 'yes' because the created ServiceAccount has permission to 'list' pods in the namespace
kubectl auth can-i list pods --namespace ${AppNamespace} --as system:serviceaccount:${AppNamespace}:${ServiceAccountName}

# Should return 'yes' because the created ServiceAccount has permission to 'delete' pods in the namespace
kubectl auth can-i delete pods --namespace ${AppNamespace} --as system:serviceaccount:${AppNamespace}:${ServiceAccountName}

# Should return 'no' because the created ServiceAccount hasn't permission to 'create' pods in the namespace
kubectl auth can-i create pods --namespace ${AppNamespace} --as system:serviceaccount:${AppNamespace}:${ServiceAccountName}

Deploy the CronJob

  • The cron schedule expressions “0 0 * * *” means to execute it everyday at 00:00.
kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: CronJob
metadata:
  name: delete-${AppName}-pods
  namespace: ${AppNamespace}
spec:
  jobTemplate:
    metadata:
      name: delete-${AppName}-pods
    spec:
      template:
        spec:
          serviceAccountName: ${ServiceAccountName}
          containers:
            - command:
                - kubectl
                - delete
                - pods
                - --selector=app=${AppName}
              image: bitnami/kubectl:1.31.3
              name: delete-${AppName}-pods
              resources: {}
          restartPolicy: OnFailure
  schedule: '0 0 * * *'
EOF

Stay tuned for my next blog post, I will show you how to do the same but using Helm instead of only Kubectl commands.