Maybe some of you use Helm to make a new release of your application hosted on Kubernetes. Maybe some of you use just simple yaml manifests because Helm is too much for you. It doesn’t matter in our today topic, as this problem can be faced independently on what tool you use. In most cases, when you update ONLY a configmap, your pods won’t recognize that (especially when the content of the configmap is loaded to memory during pod start). One of the old ways to handle that is to create new configmap, add a new ID to your pods’ definition and then remove old configmap. But it’s very ugly, terrible and inconsistent with any automate-everything mindset. But no worry – there is at least one way how you can force pods to restart in such a situation!
Force pods to restart? But why?
Kubernetes doesn’t know (at least according to my current knowledge, when I’m writing this article) how to inform pods, that configmap has been changed (in most cases). Of course, nginx-ingress is able to use an updated SSL certificate, even if it’s the same configmap. But if you have some custom configuration, dedicated to your application, there can be a problem. You cannot just update the content of the existing configmap in your Helm values files and make a new release. Pods won’t be restarted and the rolling update won’t happen. It’s very annoying especially when your application loads data from configmap to memory and you cannot do anything without restarting pods.
Kubernetes has a possibility to inform Deployment about changes in configmap, but it’s not been implemented yet – see this GitHub issue. As I mentioned before, one of the solutions is to create a completely new configmap with new data and tell Deployment to use the new configmap. Well, I’m sure you wouldn’t read this article if you thought it’s a great idea. To be honest – we need to force pods to restart, but we can do that without any service interruptions. You know, in a classic rolling update way.
How to do that?
And now there we have our most liked sentence in IT – it depends. It depends on what version of kubectl you have.
I have 1.15 or higher
Then you can just use a special command:
|
1 |
kubectl rollout restart RESOURCE/NAME |
For example:
|
1 |
kubectl rollout restart deployment/myapp |
Of course under the RESOURCE you can have a statefulSets or daemonSet. You have to have a kubectl 1.15 version to use this feature. Once you run this command, the whole pods under the controller you chose (deployment, statefulSet, daemonSet etc.) will be restarted, but in a safe, rolling-update way. That means, that you don’t need to worry about any downtime.
⚠️Please bear in mind, that version of your kubectl is important – you can use this way against older clusters! But as you can guess, having kubectl and Kubernetes out of sync is not a good idea. ⚠️
I have lower than 1.15
So here we have multiple ways to force pods to restart, but each way has its own pros and cons. I saw many bash scripts that allow us to do that, but I’d prefer a simpler, one-line way. That way:
|
1 2 |
kubectl patch deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE_NAME} -p \ "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" |
So, for example:
|
1 2 |
kubectl patch deployment myapp -n mynamespace -p \ "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"`date +'%s'`\"}}}}}" |
It will just add an annotation to your pods, and that will trigger an update. Do you know why I chose exactly THAT way, not another? Please take a look at this snippet:
|
1 2 3 4 5 6 7 8 9 |
case *extensionsv1beta1.Deployment: if obj.Spec.Paused { return nil, errors.New("can't restart paused deployment (run rollout resume first)") } if obj.Spec.Template.ObjectMeta.Annotations == nil { obj.Spec.Template.ObjectMeta.Annotations = make(map[string]string) } obj.Spec.Template.ObjectMeta.Annotations["kubectl.kubernetes.io/restartedAt"] = time.Now().Format(time.RFC3339) return runtime.Encode(scheme.Codecs.LegacyCodec(extensionsv1beta1.SchemeGroupVersion), obj) |
Even if you’re not familiar with the development, you can quickly find keywords like “Spec.Template”, “ObjectMeta.Annotations” or typical for annotations in yaml “kubectl.kubernetes.io/restartedAt”. Do you know what is it? Source code of the kubectl and its helper for pods restart. Yes, magic kubectl rollout restart just adds an annotation with a date! To be honest, it’s more professional than a simple “date”, but hey! It’s still the same (almost)!
So you can just add this annotation and everything will work. You can, of course, try to add such annotation to your Helm Charts or just Kubernetes manifests to force pods to restart every time, but I’m not sure if that’s needed for normal work. Configmaps are not changed so often.