Playing with Microclimate on IBM Cloud Private
it took me a while, and a lot of PD, but I was able to successfully resolve my issue with this: -
kubectl get pods -n micro-climate
NAME READY STATUS RESTARTS AGE
microclimate-ibm-microclimate-67cfd99c7b-zspb2 1/1 Running 0 18m
microclimate-ibm-microclimate-atrium-7f75d754fd-t97m7 1/1 Running 0 19m
microclimate-ibm-microclimate-devops-5b88cf9bcc-fr7jg 1/1 Running 0 18m
microclimate-jenkins-755d769675-jp9fj 0/1 Init:CrashLoopBackOff 8 19m
kubectl describe pod microclimate-jenkins-755d769675-jp9fj -n micro-climate
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned micro-climate/microclimate-jenkins-755d769675-jp9fj to 10.51.4.37
Normal Started 18m (x4 over 20m) kubelet, 10.51.4.37 Started container
Normal Pulling 17m (x5 over 20m) kubelet, 10.51.4.37 pulling image "ibmcom/microclimate-jenkins:1812"
Normal Pulled 17m (x5 over 20m) kubelet, 10.51.4.37 Successfully pulled image "ibmcom/microclimate-jenkins:1812"
Normal Created 17m (x5 over 20m) kubelet, 10.51.4.37 Created container
Warning BackOff 3s (x81 over 19m) kubelet, 10.51.4.37 Back-off restarting failed container
I dug into the Docker logs on one of my ICP worker nodes, using the command: -
docker logs 7ed80b2f7174 -f
...
cp: cannot create regular file '/var/jenkins_home/config.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml': Permission denied
/var/jenkins_config/apply_config.sh: 12: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/config1.xml: Permission denied
cp: cannot stat '/var/jenkins_home/config1.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/jenkins.CLI.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml': Permission denied
/var/jenkins_config/apply_config.sh: 18: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/secret.yaml: Permission denied
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: the path "/var/jenkins_home/secret.yaml" does not exist
mkdir: cannot create directory ‘/var/jenkins_home/users’: Permission denied
cp: cannot create regular file '/var/jenkins_home/users/admin/config.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/plugins.txt': Permission denied
cat: /var/jenkins_home/plugins.txt: No such file or directory
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...
WAR bundled plugins:
Installed plugins:
*:
Cleaning up locks
rm: cannot remove '/usr/share/jenkins/ref/plugins/*.lock': No such file or directory
cp: cannot stat '/usr/share/jenkins/ref/plugins/*': No such file or directory
root@dmhicp-worker-3:~# docker logs 7ed80b2f7174 -f
cp: cannot create regular file '/var/jenkins_home/config.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml': Permission denied
/var/jenkins_config/apply_config.sh: 12: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/config1.xml: Permission denied
cp: cannot stat '/var/jenkins_home/config1.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/jenkins.CLI.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml': Permission denied
/var/jenkins_config/apply_config.sh: 18: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/secret.yaml: Permission denied
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: the path "/var/jenkins_home/secret.yaml" does not exist
mkdir: cannot create directory ‘/var/jenkins_home/users’: Permission denied
cp: cannot create regular file '/var/jenkins_home/users/admin/config.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/plugins.txt': Permission denied
cat: /var/jenkins_home/plugins.txt: No such file or directory
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...
WAR bundled plugins:
Installed plugins:
*:
Cleaning up locks
rm: cannot remove '/usr/share/jenkins/ref/plugins/*.lock': No such file or directory
cp: cannot stat '/usr/share/jenkins/ref/plugins/*': No such file or directory
root@dmhicp-worker-3:~# docker logs 7ed80b2f7174 -f
cp: cannot create regular file '/var/jenkins_home/config.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml': Permission denied
/var/jenkins_config/apply_config.sh: 12: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/config1.xml: Permission denied
cp: cannot stat '/var/jenkins_home/config1.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/jenkins.CLI.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml': Permission denied
/var/jenkins_config/apply_config.sh: 18: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/secret.yaml: Permission denied
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: the path "/var/jenkins_home/secret.yaml" does not exist
mkdir: cannot create directory ‘/var/jenkins_home/users’: Permission denied
cp: cannot create regular file '/var/jenkins_home/users/admin/config.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/plugins.txt': Permission denied
cat: /var/jenkins_home/plugins.txt: No such file or directory
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...
WAR bundled plugins:
Installed plugins:
*:
Cleaning up locks
rm: cannot remove '/usr/share/jenkins/ref/plugins/*.lock': No such file or directory
cp: cannot stat '/usr/share/jenkins/ref/plugins/*': No such file or directory
...
which was strange, given that I was using a NFS v4 service ( on the Boot node of my ICP cluster ) to host the required file-systems ( as per the previous post, using Persistent Volumes and Persistent Volume Claims ).
Thankfully, a colleague had hit the same problem, albeit NOT using NFS, and the solution was, as one might imagine, permissions :-)
Yes, even though I'm running everything as root (!), and had exported the file-systems via /etc/exports : -
/export/CAM_logs *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/CAM_db *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/CAM_terraform *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/CAM_BPD_appdata *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/MC_jenkins *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/MC_microclimate *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
I needed to ensure that the underlying file-system permissions were also correct.
So this was what I had: -
ls -altrc /export
total 12
drwxr-xr-x 23 root root 4096 Jan 2 11:34 ..
drwxr-xr-x 3 root root 36 Jan 2 16:48 CAM_terraform
drwxr-xr-x 19 root root 4096 Jan 2 16:48 CAM_logs
drwxr-xr-x 5 root root 56 Jan 2 16:48 CAM_BPD_appdata
drwxr-xr-x 2 root root 6 Jan 15 16:19 MC_jenkins
drwxr-xr-x 8 root root 121 Jan 15 16:20 .
drwxr-xr-x 3 root root 25 Jan 15 16:28 MC_microclimate
drwxr-xr-x 4 999 root 4096 Jan 21 10:30 CAM_db
and this is what I needed to do: -
chmod -R 777 /export/MC_jenkins/
giving me this: -
drwxr-xr-x 3 root root 36 Jan 2 16:48 CAM_terraform
drwxr-xr-x 19 root root 4096 Jan 2 16:48 CAM_logs
drwxr-xr-x 5 root root 56 Jan 2 16:48 CAM_BPD_appdata
drwxr-xr-x 3 root root 25 Jan 15 16:28 MC_microclimate
drwxr-xr-x 4 999 root 4096 Jan 21 10:30 CAM_db
drwxrwxrwx 2 root root 6 Jan 21 15:34 MC_jenkins
In other words, I needed to set the group and world permissions to write as well as the user, thus changing FROM 755 TO 777.
As soon as I did this, the ICP ( Kubernetes ) Replica Set automatically spun up container instances on the Worker nodes which happily grabbed the storage, and I immediately saw stuff being written by Jenkins: -
ls -al /export/MC_jenkins/
total 84
drwxrwxrwx 17 root root 4096 Jan 21 15:36 .
drwxr-xr-x 8 root root 121 Jan 15 16:20 ..
drwxr-xr-x 3 fyre lpadmin 24 Jan 21 15:35 .cache
-rw-r--r-- 1 fyre lpadmin 6997 Jan 21 15:35 config1.xml
-rw-r--r-- 1 fyre lpadmin 7645 Jan 21 15:36 config.xml
-rw-r--r-- 1 fyre lpadmin 2640 Jan 21 15:35 copy_reference_file.log
drwxr-xr-x 3 fyre lpadmin 20 Jan 21 15:36 .groovy
-rw-r--r-- 1 fyre lpadmin 156 Jan 21 15:36 hudson.model.UpdateCenter.xml
-rw-r--r-- 1 fyre lpadmin 370 Jan 21 15:36 hudson.plugins.git.GitTool.xml
-rw------- 1 fyre lpadmin 1712 Jan 21 15:36 identity.key.enc
drwxr-xr-x 2 fyre lpadmin 41 Jan 21 15:35 init.groovy.d
drwxr-xr-x 3 fyre lpadmin 19 Jan 21 15:35 .java
-rw-r--r-- 1 fyre lpadmin 94 Jan 21 15:35 jenkins.CLI.xml
-rw-r--r-- 1 fyre lpadmin 5 Jan 21 15:36 jenkins.install.InstallUtil.lastExecVersion
-rw-r--r-- 1 fyre lpadmin 274 Jan 21 15:35 jenkins.model.JenkinsLocationConfiguration.xml
drwxr-xr-x 2 fyre lpadmin 6 Jan 21 15:36 jobs
drwxr-xr-x 4 fyre lpadmin 37 Jan 21 15:35 .kube
drwxr-xr-x 3 fyre lpadmin 19 Jan 21 15:36 logs
-rw-r--r-- 1 fyre lpadmin 907 Jan 21 15:36 nodeMonitors.xml
drwxr-xr-x 2 fyre lpadmin 6 Jan 21 15:36 nodes
-rw-r--r-- 1 fyre lpadmin 1034 Jan 21 15:35 org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml
drwxr-xr-x 56 fyre lpadmin 12288 Jan 21 15:36 plugins
-rw-r--r-- 1 fyre lpadmin 24 Jan 21 15:35 plugins.txt
-rw-r--r-- 1 fyre lpadmin 64 Jan 21 15:35 secret.key
-rw-r--r-- 1 fyre lpadmin 0 Jan 21 15:35 secret.key.not-so-secret
drwxr-xr-x 4 fyre lpadmin 263 Jan 21 15:36 secrets
-rw-r--r-- 1 fyre lpadmin 0 Jan 21 15:35 secret.yaml
drwxr-xr-x 2 fyre lpadmin 67 Jan 21 15:36 updates
drwxr-xr-x 2 fyre lpadmin 24 Jan 21 15:36 userContent
drwxr-xr-x 3 fyre lpadmin 19 Jan 21 15:35 users
drwxr-xr-x 11 fyre lpadmin 4096 Jan 21 15:35 war
drwxr-xr-x 2 fyre lpadmin 6 Jan 21 15:36 workflow-libs
with happy containers: -
docker logs 7f347b0a46e2 -f
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: no objects passed to create
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...
Downloading plugin: credentials-binding from https://updates.jenkins.io/download/plugins/credentials-binding/1.16/credentials-binding.hpi
> credentials-binding depends on workflow-step-api:2.10,credentials:2.1.7,plain-credentials:1.3,ssh-credentials:1.11,structs:1.7
Downloading plugin: workflow-step-api from https://updates.jenkins.io/download/plugins/workflow-step-api/latest/workflow-step-api.hpi
Downloading plugin: credentials from https://updates.jenkins.io/download/plugins/credentials/latest/credentials.hpi
Downloading plugin: plain-credentials from https://updates.jenkins.io/download/plugins/plain-credentials/latest/plain-credentials.hpi
Downloading plugin: ssh-credentials from https://updates.jenkins.io/download/plugins/ssh-credentials/latest/ssh-credentials.hpi
Downloading plugin: structs from https://updates.jenkins.io/download/plugins/structs/latest/structs.hpi
> credentials depends on structs:1.7
> workflow-step-api depends on structs:1.5
> plain-credentials depends on credentials:2.1.16
> ssh-credentials depends on credentials:2.1.17
WAR bundled plugins:
Installed plugins:
credentials-binding:1.16
credentials:2.1.18
plain-credentials:1.5
ssh-credentials:1.14
structs:1.17
workflow-step-api:2.18
Cleaning up locks
root@dmhicp-worker-3:~#
root@dmhicp-worker-3:~# docker logs 7f347b0a46e2 -f
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: no objects passed to create
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...
Downloading plugin: credentials-binding from https://updates.jenkins.io/download/plugins/credentials-binding/1.16/credentials-binding.hpi
> credentials-binding depends on workflow-step-api:2.10,credentials:2.1.7,plain-credentials:1.3,ssh-credentials:1.11,structs:1.7
Downloading plugin: workflow-step-api from https://updates.jenkins.io/download/plugins/workflow-step-api/latest/workflow-step-api.hpi
Downloading plugin: credentials from https://updates.jenkins.io/download/plugins/credentials/latest/credentials.hpi
Downloading plugin: plain-credentials from https://updates.jenkins.io/download/plugins/plain-credentials/latest/plain-credentials.hpi
Downloading plugin: ssh-credentials from https://updates.jenkins.io/download/plugins/ssh-credentials/latest/ssh-credentials.hpi
Downloading plugin: structs from https://updates.jenkins.io/download/plugins/structs/latest/structs.hpi
> credentials depends on structs:1.7
> workflow-step-api depends on structs:1.5
> plain-credentials depends on credentials:2.1.16
> ssh-credentials depends on credentials:2.1.17
WAR bundled plugins:
Installed plugins:
credentials-binding:1.16
credentials:2.1.18
plain-credentials:1.5
ssh-credentials:1.14
structs:1.17
workflow-step-api:2.18
Cleaning up locks
and all of the Microclimate pods started playing nicely: -
kubectl get pods -n micro-climate
NAME READY STATUS RESTARTS AGE
microclimate-ibm-microclimate-67cfd99c7b-66jj4 1/1 Running 0 29m
microclimate-ibm-microclimate-atrium-7f75d754fd-txn5p 1/1 Running 0 29m
microclimate-ibm-microclimate-devops-568c4c5989-prcbr 1/1 Running 0 29m
microclimate-jenkins-678584959-9hqld 1/1 Running 0 29m
and a working Microclimate environment .....
No comments:
Post a Comment