One of my colleagues saw an interesting issue when trying to build a new Kubernetes cluster on an Ubuntu Linux environment ( on IBM Z ).
For the record, we're running Kubernetes inside Ubuntu containers which are hosted, via runq, on a Secure Service Container (SSC) logical partition (LPAR). In this scenario, we're using docker as the container runtime inside the Ubuntu container which is running inside the SSC LPAR ( nested FTW ).
However, the specific issue seen when running commands such as: -
kubeadm init --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors=all
wasn't directly related to the use of runq or the SSC LPAR.
Instead, the command returned: -
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/pause:3.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/etcd:3.4.13-0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/coredns:1.7.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
Interestingly, an article on LinkedIn led me to the solution: -
In part, the author had me reload/restart the Docker service: -
service docker reload
service docker restart
Prior to this, I'd also checked some of the missing pre-requisite steps: -
apt-get install -y ca-certificates gnupg2 curl apt-transport-https
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
plus opening up some firewall ports, using iptables : -
iptables -A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 10250 -j ACCEPT
However, I suspect that the combination of: -
apt-get install -y ca-certificates gnupg2 curl apt-transport-https
and: -
service docker reload
service docker restart
did the trick.
One nice thing that I learned ( TIL ) was: -
kubeadm config images pull
to test whether kubeadm can get it's required image before starting the init process: -
I0601 10:02:47.536650 25480 version.go:251] remote version is much newer: v1.21.1; falling back to: stable-1.20
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.20.7
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.20.7
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.20.7
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.20.7
[config/images] Pulled k8s.gcr.io/pause:3.2
[config/images] Pulled k8s.gcr.io/etcd:3.4.13-0
[config/images] Pulled k8s.gcr.io/coredns:1.7.0
Having pulled the images normally, without any trust exceptions, kubeadm init ran happily .....
