Building Kubernetes on Linux on IBM Z - it's a matter of trust ...

 One of my colleagues saw an interesting issue when trying to build a new Kubernetes cluster on an Ubuntu Linux environment ( on IBM Z ).

For the record, we're running Kubernetes inside Ubuntu containers which are hosted, via runq, on a Secure Service Container (SSC) logical partition (LPAR). In this scenario, we're using docker as the container runtime inside the Ubuntu container which is running inside the SSC LPAR ( nested FTW ).

However, the specific issue seen when running commands such as: -

kubeadm init --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors=all

wasn't directly related to the use of runq or the SSC LPAR.

Instead, the command returned: -

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/pause:3.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/etcd:3.4.13-0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/coredns:1.7.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1

Interestingly, an article on LinkedIn led me to the solution: -

Fixing the output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority

In part, the author had me reload/restart the Docker service: -

service docker reload

service docker restart

Prior to this, I'd also checked some of the missing pre-requisite steps: -

apt-get install -y ca-certificates gnupg2 curl apt-transport-https

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

plus opening up some firewall ports, using iptables : -

iptables -A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT

iptables -A INPUT -p tcp -m tcp --dport 10250 -j ACCEPT

However, I suspect that the combination of: -

apt-get install -y ca-certificates gnupg2 curl apt-transport-https

and: -

service docker reload

service docker restart

did the trick.

One nice thing that I learned ( TIL ) was: -

kubeadm config images pull

to test whether kubeadm can get it's required image before starting the init process: -

I0601 10:02:47.536650   25480 version.go:251] remote version is much newer: v1.21.1; falling back to: stable-1.20

[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.20.7

[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.20.7

[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.20.7

[config/images] Pulled k8s.gcr.io/kube-proxy:v1.20.7

[config/images] Pulled k8s.gcr.io/pause:3.2

[config/images] Pulled k8s.gcr.io/etcd:3.4.13-0

[config/images] Pulled k8s.gcr.io/coredns:1.7.0

Having pulled the images normally, without any trust exceptions, kubeadm init ran happily .....