Helm Charts and Docker image version issue

Helm is a tool that streamlines installing and managing Kubernetes applications. It is kind of a OS package manager like apt/yum/homebrew. In Helm we have term of Charts which are packages of pre-configured Kubernetes resources. Using these charts we can easily deploy the applications to our Kubernetes cluster.

Helm also maintains a list of stable charts on their Github repository. We can deploy them with a single command helm install stable/<chart-name>.

The Issue

I was working on the deployment of Loki stack few days ago. Grafana is one of the component that I needed. I used the following helm install command as usual to deploy Grafana to my cluster.

$ helm install stable/grafana -n loki-grafana

The above command ran successfully. However, the deployment READY status kept stucking at 0/1.

$ kubectl get deployment loki-grafana
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
loki-grafana   0/1     1            1           15m

By looking at its pod information, I found the pod’s Event recorded several errors related to the image pulling.

$ kubectl describe pod loki-grafana-5b9fdc85b8-rj9rt
...
Events:
  Type     Reason     Age                From                 Message
  ----     ------     ----               ----                 -------
  Normal   Scheduled  54s                default-scheduler    Successfully assigned default/loki-grafana-5b9fdc85b8-rj9rt to k8s-w5
  Normal   BackOff    21s (x2 over 50s)  kubelet, k8s-w5      Back-off pulling image "grafana/grafana:6.2.3"
  Warning  Failed     21s (x2 over 50s)  kubelet, k8s-w5      Error: ImagePullBackOff
  Normal   Pulling    9s (x3 over 54s)   kubelet, k8s-w5      Pulling image "grafana/grafana:6.2.3"
  Warning  Failed     6s (x3 over 50s)   kubelet, k8s-w5      Failed to pull image "grafana/grafana:6.2.3": rpc error: code = Unknown desc = Error response from daemon: manifest for grafana/grafana:6.2.3 not found
  Warning  Failed     6s (x3 over 50s)   kubelet, k8s-w5      Error: ErrImagePull

Somehow the grafana/grafana:6.2.3 image was not found. But it should be. This is stable chart release which will fetch the exact Grafana image and tag from Docker Hub repository.

So I tried to pull that image manually on my local machine to verify.

$ docker pull grafana/grafana:6.2.3
Error response from daemon: manifest for grafana/grafana:6.2.3 not found

The same issue. It wasn’t available for my local Docker instance as well. Ok, final check, I went to the Grafana image tags on Docker Hub and recognized that the tag 6.2.3 was really missing.

The Resolution

So the tag was missing on the Docker Hub although Grafana Labs had made a release for their software and updated the Helm Chart. So the solution was to use the older version of Grafana (older tag) instead of 6.2.3.

Back to the Grafana helm chart repository on Github, I found the config value need to be added, it is image.tag which help to specify the Docker image tag we want to use during the Helm Chart installation.

$ helm install stable/grafana -n loki-grafana --set "image.tag=6.2.2"

Double check the pod’s event log, it is working now!

Events:
  Type    Reason     Age   From                 Message
  ----    ------     ----  ----                 -------
  Normal  Scheduled  30s   default-scheduler    Successfully assigned default/loki-grafana-d577d745c-jh4ps to k8s-w5
  Normal  Pulled     29s   kubelet, k8s-w5      Container image "grafana/grafana:6.2.2" already present on machine
  Normal  Created    29s   kubelet, k8s-w5      Created container grafana
  Normal  Started    29s   kubelet, k8s-w5      Started container grafana