Tuesday, 1 June 2021

Building Kubernetes on Linux on IBM Z - it's a matter of trust ...

 One of my colleagues saw an interesting issue when trying to build a new Kubernetes cluster on an Ubuntu Linux environment ( on IBM Z ).

For the record, we're running Kubernetes inside Ubuntu containers which are hosted, via runq, on a Secure Service Container (SSC) logical partition (LPAR). In this scenario, we're using docker as the container runtime inside the Ubuntu container which is running inside the SSC LPAR ( nested FTW ).

However, the specific issue seen when running commands such as: -

kubeadm init --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors=all

wasn't directly related to the use of runq or the SSC LPAR.

Instead, the command returned: -

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.20.7: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/pause:3.2: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/etcd:3.4.13-0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1
[WARNING ImagePull]: failed to pull image k8s.gcr.io/coredns:1.7.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: x509: certificate signed by unknown authority
, error: exit status 1

Interestingly, an article on LinkedIn led me to the solution: -


In part, the author had me reload/restart the Docker service: -

service docker reload
service docker restart

Prior to this, I'd also checked some of the missing pre-requisite steps: -

apt-get install -y ca-certificates gnupg2 curl apt-transport-https

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

plus opening up some firewall ports, using iptables : -

iptables -A INPUT -p tcp -m tcp --dport 6443 -j ACCEPT
iptables -A INPUT -p tcp -m tcp --dport 10250 -j ACCEPT

However, I suspect that the combination of: -

apt-get install -y ca-certificates gnupg2 curl apt-transport-https

and: -

service docker reload
service docker restart

did the trick.

One nice thing that I learned ( TIL ) was: -

kubeadm config images pull

to test whether kubeadm can get it's required image before starting the init process: -

I0601 10:02:47.536650   25480 version.go:251] remote version is much newer: v1.21.1; falling back to: stable-1.20
[config/images] Pulled k8s.gcr.io/kube-apiserver:v1.20.7
[config/images] Pulled k8s.gcr.io/kube-controller-manager:v1.20.7
[config/images] Pulled k8s.gcr.io/kube-scheduler:v1.20.7
[config/images] Pulled k8s.gcr.io/kube-proxy:v1.20.7
[config/images] Pulled k8s.gcr.io/pause:3.2
[config/images] Pulled k8s.gcr.io/etcd:3.4.13-0
[config/images] Pulled k8s.gcr.io/coredns:1.7.0

Having pulled the images normally, without any trust exceptions, kubeadm init ran happily .....

No comments:

Yay, VMware Fusion and macOS Big Sur - no longer "NAT good friends" - forgive the double negative and the terrible pun ...

After macOS 11 Big Sur was released in 2020, VMware updated their Fusion product to v12 and, sadly, managed to break Network Address Trans...