Monday, 31 December 2018

Docker Secrets - And there's more ....

Hot on the heels of my last post: -

Shush, it's a secret .... 

having created my Docker secret: -

   kubectl create secret docker-registry --docker-username= --docker-password= --docker-email= -n services

as per this: -

Creating Docker Store secret

I made a slight mistake ....

For the --docker-username parameter, I used my email address - with which I can log into the Docker Hub: -


*BUT* this proved to be a bad idea.

Whilst building out my IBM Cloud Automation Manager (CAM) environment, I saw this: -

kubectl get -n services pods

NAME                                        READY     STATUS              RESTARTS   AGE
cam-bpd-cds-5f57588776-jnw8c                0/1       Init:ErrImagePull   0          58s
cam-bpd-mariadb-6b98577f65-g75mj            0/1       Pending             0          58s
cam-bpd-mds-69f5d6988c-tfdrh                0/1       Init:ErrImagePull   0          58s
cam-bpd-ui-6c86d7d6f7-wm22d                 0/1       Pending             0          57s
cam-broker-7b86c6cff5-wcwd9                 0/1       Init:ErrImagePull   0          56s
cam-iaas-7884798b9-ztvmg                    0/1       Init:ErrImagePull   0          56s
cam-mongo-55c5976cf5-p5xtb                  0/1       ErrImagePull        0          55s
cam-orchestration-664f9647d8-sgxv2          0/1       Init:ErrImagePull   0          54s
cam-portal-ui-858b7dfcbd-qxd4k              0/1       Init:ErrImagePull   0          53s
cam-provider-helm-84b6fd45c6-zt8mt          0/1       Init:ErrImagePull   0          52s
cam-provider-terraform-79794ff875-82k4b     0/1       Init:ErrImagePull   0          52s
cam-proxy-5d64b478d6-68whm                  0/1       Init:ErrImagePull   0          51s
cam-service-composer-api-64ff5c747c-ctq5g   0/1       Init:ErrImagePull   0          49s
cam-service-composer-ui-54799748fb-dkpfn    0/1       Init:ErrImagePull   0          48s
cam-tenant-api-84f9bc79d-lsgvp              0/1       Pending             0          48s
cam-ui-basic-7d9ffc5858-b7qkr               0/1       Pending             0          47s
cam-ui-connections-6cdb6cf45b-wcw69         0/1       Pending             0          46s
cam-ui-instances-54f85dfbd4-xx2zd           0/1       Pending             0          45s
cam-ui-templates-57b467554f-l6nmh           0/1       Pending             0          44s
redis-74b9dc6d48-l4g5d                      0/1       ErrImagePull        0          50s

and, when I drilled into one of the failing pods via the IBM Cloud Private (ICP) web UI, I saw this: -

Failed to pull image "store/ibmcorp/icam-busybox:3.1.0.0-x86_64": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/store/ibmcorp/icam-busybox/manifests/3.1.0.0-x86_64: unauthorized: incorrect username or password

I validated the exception via the CLI: -

docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: david_hay@uk.ibm.com
Password: 
Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password

I then re-read the documentation AND the prompt from docker login again, and realised that I was using my email address RATHER than my username.

Once I used my username: -

docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.
Username: davidhay1969
Password: 
Login Succeeded

Back in the ICP world, I deleted my secret: -

kubectl delete secret david-hay

and recreated it using the username rather than the email address.

I then killed the failing pod deployments: -

kubectl delete pod cam-broker-7b86c6cff5-wcwd9
kubectl delete pod cam-bpd-cds-5f57588776-jnw8c
kubectl delete pod cam-bpd-mariadb-6b98577f65-g75mj
kubectl delete pod cam-bpd-mds-69f5d6988c-tfdrh
kubectl delete pod cam-bpd-ui-6c86d7d6f7-wm22d
kubectl delete pod cam-iaas-7884798b9-ztvmg
kubectl delete pod cam-log-rotation-1546259700-zgx5v
kubectl delete pod cam-mongo-55c5976cf5-p5xtb
kubectl delete pod cam-orchestration-664f9647d8-sgxv2
kubectl delete pod cam-portal-ui-858b7dfcbd-qxd4k
kubectl delete pod cam-provider-helm-84b6fd45c6-zt8mt
kubectl delete pod cam-provider-terraform-79794ff875-82k4b
kubectl delete pod cam-proxy-5d64b478d6-68whm
kubectl delete pod cam-service-composer-api-64ff5c747c-ctq5g
kubectl delete pod cam-service-composer-ui-54799748fb-dkpfn
kubectl delete pod cam-tenant-api-84f9bc79d-lsgvp
kubectl delete pod cam-ui-basic-7d9ffc5858-b7qkr
kubectl delete pod cam-ui-connections-6cdb6cf45b-wcw69
kubectl delete pod cam-ui-instances-54f85dfbd4-xx2zd
kubectl delete pod cam-ui-templates-57b467554f-l6nmh

which then "forced" the ICP / Kubernetes Replica Set to recreate the pods.

Now I see this: -

kubectl get -n services pods

NAME                                        READY     STATUS    RESTARTS   AGE
cam-bpd-cds-5f57588776-fzq92                1/1       Running   0          28m
cam-bpd-mariadb-6b98577f65-qj7kf            0/1       Pending   0          28m
cam-bpd-mds-69f5d6988c-z9m6q                1/1       Running   0          28m
cam-bpd-ui-6c86d7d6f7-kxdrc                 0/1       Pending   0          28m
cam-broker-7b86c6cff5-kxz9r                 0/1       Pending   0          31m
cam-iaas-7884798b9-bcnhd                    1/1       Running   0          28m
cam-log-rotation-1546263600-w62hj           0/1       Pending   0          16m
cam-mongo-55c5976cf5-q5b9s                  1/1       Running   0          28m
cam-orchestration-664f9647d8-r7qxv          1/1       Running   0          28m
cam-portal-ui-858b7dfcbd-wwshz              1/1       Running   0          27m
cam-provider-helm-84b6fd45c6-rb5cl          1/1       Running   0          27m
cam-provider-terraform-79794ff875-6hh64     0/1       Pending   0          27m
cam-proxy-5d64b478d6-kwpg5                  1/1       Running   0          27m
cam-service-composer-api-64ff5c747c-2d6v9   1/1       Running   0          27m
cam-service-composer-ui-54799748fb-424lg    1/1       Running   0          27m
cam-tenant-api-84f9bc79d-bpnf5              1/1       Running   0          26m
cam-ui-basic-7d9ffc5858-drrh8               0/1       Pending   0          26m
cam-ui-connections-6cdb6cf45b-465tk         0/1       Pending   0          26m
cam-ui-instances-54f85dfbd4-r7kfv           1/1       Running   0          25m
cam-ui-templates-57b467554f-wfmpc           0/1       Pending   0          25m
redis-74b9dc6d48-l4g5d                      1/1       Running   0          1h

which is much much better.

Every day, it's a school day !

No comments:

SSH and "Too many authentication failures" - a new one on me

Having created a new user on an Ubuntu 16.04 boxen, I started seeing this: - Received disconnect from 192.168.3.123 port 22:2: Too many au...