Thursday, 11 February 2021

Gah, ImagePullBackOff with Calico CNI running on Kubernetes

Whilst deploying Calico Node etc. to my cluster, via: -

kubectl apply -f calico.yaml

and whilst checking the running Pods, via: -

kubectl get pods -A

I was seeing: -

...

kube-system   calico-node-9srv6                         0/1     Init:ErrImagePull   0          8s

...

I dug into that failing Pod with: -

kubectl describe pod calico-node-9srv6 --namespace kube-system

which showed: -

...
  Type     Reason   Age                    From                   Message
  ----     ------   ----                   ----                   -------
  Normal   BackOff  17m (x303 over 88m)    kubelet, 50c933ad26be  Back-off pulling image "us.icr.io/davehay/calico/cni:latest-s390x"
  Warning  Failed   2m56s (x368 over 88m)  kubelet, 50c933ad26be  Error: ImagePullBackOff
...


Now I knew that it wasn't an authentication issue, as my YAML was also defining a Secret, as per my previous post: -


and had defined that Secret within the YAML, using: -

...
imagePullSecrets:
- name: my_secret
...

So why wasn't it working .... ?

And then it struck me .... DOH!

My Pod is running inside the kube-system Namespace ....

You know where I'm going with this, am I right ?

Yes, my Secret was NOT inside the same kube-system Namespace, but was in default.

Once I updated my YAML to redefine my Secret: -

...
apiVersion: v1
kind: Secret
data:
  .dockerconfigjson:
    <HERE'S THE BASE64 ENCODED STUFF>
metadata:
  name: my_secret
  namespace: kube-system
type: kubernetes.io/dockerconfigjson
...

re-applied the YAML, and deleted the failing Pod, all was well

No comments:

Note to self - use kubectl to query images in a pod or deployment

In both cases, we use JSON ... For a deployment, we can do this: - kubectl get deployment foobar --namespace snafu --output jsonpath="{...