Thursday, 4 March 2021

Gah, again with the ImagePullBackOff

 So, following on from this: -

Gah, ImagePullBackOff with Calico CNI running on Kubernetes

I was again seeing this: -

kube-system   calico-node-lxmk4                          0/1     Init:ImagePullBackOff   0          5m26s

and, upon further digging: -

kubectl describe pod calico-node-lxmk4 --namespace kube-system

Type     Reason     Age                    From                   Message
----     ------     ----                   ----                   -------
Normal   Scheduled  5m47s                  default-scheduler      Successfully assigned kube-system/calico-node-lxmk4 to 667ceb40fc75
Normal   Pulling    4m24s (x4 over 5m46s)  kubelet, 667ceb40fc75  Pulling image "us.icr.io/mynamespace/calico/cni:v3.16.5"
Warning  Failed     4m23s (x4 over 5m45s)  kubelet, 667ceb40fc75  Failed to pull image "us.icr.io/mynamespace/calico/cni:v3.16.5": rpc error: code = Unknown desc = Error response from daemon: Get https://us.icr.io/v2/mynamespace/calico/cni/manifests/v3.16.5: unauthorized: The login credentials are not valid, or your IBM Cloud account is not active.
Warning  Failed     4m23s (x4 over 5m45s)  kubelet, 667ceb40fc75  Error: ErrImagePull
Warning  Failed     3m57s (x7 over 5m45s)  kubelet, 667ceb40fc75  Error: ImagePullBackOff
Normal   BackOff    46s (x21 over 5m45s)   kubelet, 667ceb40fc75  Back-off pulling image "us.icr.io/mynamespace/calico/cni:v3.16.5"

Note that my images are coming from IBM Container Registry, rather than Docker Hub, and that's the key .....

I was following this: -


which describes how one can generate a K8s secret from an existing docker login by grabbing the content of ~/.docker/config.json

Therefore, I was doing this: -

kubectl create secret generic regcred --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson

having previously logged in: -

echo "<MY API KEY>" | docker login -u iamapikey --password-stdin us.icr.io

which creates/updates /root/.docker/config.json

And that's where I was failing .....

Finally, after a few hours of head-banging, I looked back through my notes and realised that, for previous activities, including Tekton Pipelines / Triggers, I used a different approach to generate the secret: -

kubectl create secret docker-registry regcred --namespace kube-system --docker-server='https://us.icr.io' --docker-username='iamapikey' --docker-password='<MY API KEY>'

And, of course, it worked .....

Every day is ......

No comments:

Yay, VMware Fusion and macOS Big Sur - no longer "NAT good friends" - forgive the double negative and the terrible pun ...

After macOS 11 Big Sur was released in 2020, VMware updated their Fusion product to v12 and, sadly, managed to break Network Address Trans...