Thursday, 11 February 2021

Gah, ImagePullBackOff with Calico CNI running on Kubernetes

Whilst deploying Calico Node etc. to my cluster, via: -

kubectl apply -f calico.yaml

and whilst checking the running Pods, via: -

kubectl get pods -A

I was seeing: -

...

kube-system   calico-node-9srv6                         0/1     Init:ErrImagePull   0          8s

...

I dug into that failing Pod with: -

kubectl describe pod calico-node-9srv6 --namespace kube-system

which showed: -

...
  Type     Reason   Age                    From                   Message
  ----     ------   ----                   ----                   -------
  Normal   BackOff  17m (x303 over 88m)    kubelet, 50c933ad26be  Back-off pulling image "us.icr.io/davehay/calico/cni:latest-s390x"
  Warning  Failed   2m56s (x368 over 88m)  kubelet, 50c933ad26be  Error: ImagePullBackOff
...


Now I knew that it wasn't an authentication issue, as my YAML was also defining a Secret, as per my previous post: -


and had defined that Secret within the YAML, using: -

...
imagePullSecrets:
- name: my_secret
...

So why wasn't it working .... ?

And then it struck me .... DOH!

My Pod is running inside the kube-system Namespace ....

You know where I'm going with this, am I right ?

Yes, my Secret was NOT inside the same kube-system Namespace, but was in default.

Once I updated my YAML to redefine my Secret: -

...
apiVersion: v1
kind: Secret
data:
  .dockerconfigjson:
    <HERE'S THE BASE64 ENCODED STUFF>
metadata:
  name: my_secret
  namespace: kube-system
type: kubernetes.io/dockerconfigjson
...

re-applied the YAML, and deleted the failing Pod, all was well

No comments:

Yay, VMware Fusion and macOS Big Sur - no longer "NAT good friends" - forgive the double negative and the terrible pun ...

After macOS 11 Big Sur was released in 2020, VMware updated their Fusion product to v12 and, sadly, managed to break Network Address Trans...