Monday, 21 January 2019

Microclimate on IBM Cloud Private - Permission to write

Following on from my earlier post: -

 Playing with Microclimate on IBM Cloud Private 

it took me a while, and a lot of PD, but I was able to successfully resolve my issue with this: -

kubectl get pods -n micro-climate

NAME                                                    READY     STATUS                  RESTARTS   AGE
microclimate-ibm-microclimate-67cfd99c7b-zspb2          1/1       Running                 0          18m
microclimate-ibm-microclimate-atrium-7f75d754fd-t97m7   1/1       Running                 0          19m
microclimate-ibm-microclimate-devops-5b88cf9bcc-fr7jg   1/1       Running                 0          18m
microclimate-jenkins-755d769675-jp9fj                   0/1       Init:CrashLoopBackOff   8          19m

kubectl describe pod microclimate-jenkins-755d769675-jp9fj -n micro-climate

  Type     Reason     Age                From                 Message
  ----     ------     ----               ----                 -------
  Normal   Scheduled  20m                default-scheduler    Successfully assigned micro-climate/microclimate-jenkins-755d769675-jp9fj to 10.51.4.37
  Normal   Started    18m (x4 over 20m)  kubelet, 10.51.4.37  Started container
  Normal   Pulling    17m (x5 over 20m)  kubelet, 10.51.4.37  pulling image "ibmcom/microclimate-jenkins:1812"
  Normal   Pulled     17m (x5 over 20m)  kubelet, 10.51.4.37  Successfully pulled image "ibmcom/microclimate-jenkins:1812"
  Normal   Created    17m (x5 over 20m)  kubelet, 10.51.4.37  Created container
  Warning  BackOff    3s (x81 over 19m)  kubelet, 10.51.4.37  Back-off restarting failed container

I dug into the Docker logs on one of my ICP worker nodes, using the command: -

docker logs 7ed80b2f7174 -f

...
cp: cannot create regular file '/var/jenkins_home/config.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml': Permission denied
/var/jenkins_config/apply_config.sh: 12: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/config1.xml: Permission denied
cp: cannot stat '/var/jenkins_home/config1.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/jenkins.CLI.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml': Permission denied
/var/jenkins_config/apply_config.sh: 18: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/secret.yaml: Permission denied
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: the path "/var/jenkins_home/secret.yaml" does not exist
mkdir: cannot create directory ‘/var/jenkins_home/users’: Permission denied
cp: cannot create regular file '/var/jenkins_home/users/admin/config.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/plugins.txt': Permission denied
cat: /var/jenkins_home/plugins.txt: No such file or directory
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...

WAR bundled plugins:


Installed plugins:
*:
Cleaning up locks
rm: cannot remove '/usr/share/jenkins/ref/plugins/*.lock': No such file or directory
cp: cannot stat '/usr/share/jenkins/ref/plugins/*': No such file or directory
root@dmhicp-worker-3:~# docker logs 7ed80b2f7174 -f
cp: cannot create regular file '/var/jenkins_home/config.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml': Permission denied
/var/jenkins_config/apply_config.sh: 12: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/config1.xml: Permission denied
cp: cannot stat '/var/jenkins_home/config1.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/jenkins.CLI.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml': Permission denied
/var/jenkins_config/apply_config.sh: 18: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/secret.yaml: Permission denied
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: the path "/var/jenkins_home/secret.yaml" does not exist
mkdir: cannot create directory ‘/var/jenkins_home/users’: Permission denied
cp: cannot create regular file '/var/jenkins_home/users/admin/config.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/plugins.txt': Permission denied
cat: /var/jenkins_home/plugins.txt: No such file or directory
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...

WAR bundled plugins:


Installed plugins:
*:
Cleaning up locks
rm: cannot remove '/usr/share/jenkins/ref/plugins/*.lock': No such file or directory
cp: cannot stat '/usr/share/jenkins/ref/plugins/*': No such file or directory
root@dmhicp-worker-3:~# docker logs 7ed80b2f7174 -f
cp: cannot create regular file '/var/jenkins_home/config.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml': Permission denied
/var/jenkins_config/apply_config.sh: 12: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/config1.xml: Permission denied
cp: cannot stat '/var/jenkins_home/config1.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/jenkins.CLI.xml': Permission denied
cp: cannot create regular file '/var/jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml': Permission denied
/var/jenkins_config/apply_config.sh: 18: /var/jenkins_config/apply_config.sh: cannot create /var/jenkins_home/secret.yaml: Permission denied
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: the path "/var/jenkins_home/secret.yaml" does not exist
mkdir: cannot create directory ‘/var/jenkins_home/users’: Permission denied
cp: cannot create regular file '/var/jenkins_home/users/admin/config.xml': No such file or directory
cp: cannot create regular file '/var/jenkins_home/plugins.txt': Permission denied
cat: /var/jenkins_home/plugins.txt: No such file or directory
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...

WAR bundled plugins:


Installed plugins:
*:
Cleaning up locks
rm: cannot remove '/usr/share/jenkins/ref/plugins/*.lock': No such file or directory
cp: cannot stat '/usr/share/jenkins/ref/plugins/*': No such file or directory
...

which was strange, given that I was using a NFS v4 service ( on the Boot node of my ICP cluster ) to host the required file-systems ( as per the previous post, using Persistent Volumes and Persistent Volume Claims ).

Thankfully, a colleague had hit the same problem, albeit NOT using NFS, and the solution was, as one might imagine, permissions :-)

Yes, even though I'm running everything as root (!), and had exported the file-systems via /etc/exports : -

/export/CAM_logs *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/CAM_db *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/CAM_terraform *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/CAM_BPD_appdata *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/MC_jenkins *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)
/export/MC_microclimate *(rw,nohide,insecure,no_subtree_check,async,no_root_squash)

I needed to ensure that the underlying file-system permissions were also correct.

So this was what I had: -

ls -altrc /export

total 12
drwxr-xr-x 23 root root 4096 Jan  2 11:34 ..
drwxr-xr-x  3 root root   36 Jan  2 16:48 CAM_terraform
drwxr-xr-x 19 root root 4096 Jan  2 16:48 CAM_logs
drwxr-xr-x  5 root root   56 Jan  2 16:48 CAM_BPD_appdata
drwxr-xr-x  2 root root    6 Jan 15 16:19 MC_jenkins
drwxr-xr-x  8 root root  121 Jan 15 16:20 .
drwxr-xr-x  3 root root   25 Jan 15 16:28 MC_microclimate
drwxr-xr-x  4  999 root 4096 Jan 21 10:30 CAM_db

and this is what I needed to do: -

chmod -R 777 /export/MC_jenkins/

giving me this: -

drwxr-xr-x  3 root root   36 Jan  2 16:48 CAM_terraform
drwxr-xr-x 19 root root 4096 Jan  2 16:48 CAM_logs
drwxr-xr-x  5 root root   56 Jan  2 16:48 CAM_BPD_appdata
drwxr-xr-x  3 root root   25 Jan 15 16:28 MC_microclimate
drwxr-xr-x  4  999 root 4096 Jan 21 10:30 CAM_db
drwxrwxrwx  2 root root    6 Jan 21 15:34 MC_jenkins

In other words, I needed to set the group and world permissions to write as well as the user, thus changing FROM 755 TO 777.

As soon as I did this, the ICP ( Kubernetes ) Replica Set automatically spun up container instances on the Worker nodes which happily grabbed the storage, and I immediately saw stuff being written by Jenkins: -

ls -al /export/MC_jenkins/

total 84
drwxrwxrwx 17 root root     4096 Jan 21 15:36 .
drwxr-xr-x  8 root root      121 Jan 15 16:20 ..
drwxr-xr-x  3 fyre lpadmin    24 Jan 21 15:35 .cache
-rw-r--r--  1 fyre lpadmin  6997 Jan 21 15:35 config1.xml
-rw-r--r--  1 fyre lpadmin  7645 Jan 21 15:36 config.xml
-rw-r--r--  1 fyre lpadmin  2640 Jan 21 15:35 copy_reference_file.log
drwxr-xr-x  3 fyre lpadmin    20 Jan 21 15:36 .groovy
-rw-r--r--  1 fyre lpadmin   156 Jan 21 15:36 hudson.model.UpdateCenter.xml
-rw-r--r--  1 fyre lpadmin   370 Jan 21 15:36 hudson.plugins.git.GitTool.xml
-rw-------  1 fyre lpadmin  1712 Jan 21 15:36 identity.key.enc
drwxr-xr-x  2 fyre lpadmin    41 Jan 21 15:35 init.groovy.d
drwxr-xr-x  3 fyre lpadmin    19 Jan 21 15:35 .java
-rw-r--r--  1 fyre lpadmin    94 Jan 21 15:35 jenkins.CLI.xml
-rw-r--r--  1 fyre lpadmin     5 Jan 21 15:36 jenkins.install.InstallUtil.lastExecVersion
-rw-r--r--  1 fyre lpadmin   274 Jan 21 15:35 jenkins.model.JenkinsLocationConfiguration.xml
drwxr-xr-x  2 fyre lpadmin     6 Jan 21 15:36 jobs
drwxr-xr-x  4 fyre lpadmin    37 Jan 21 15:35 .kube
drwxr-xr-x  3 fyre lpadmin    19 Jan 21 15:36 logs
-rw-r--r--  1 fyre lpadmin   907 Jan 21 15:36 nodeMonitors.xml
drwxr-xr-x  2 fyre lpadmin     6 Jan 21 15:36 nodes
-rw-r--r--  1 fyre lpadmin  1034 Jan 21 15:35 org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml
drwxr-xr-x 56 fyre lpadmin 12288 Jan 21 15:36 plugins
-rw-r--r--  1 fyre lpadmin    24 Jan 21 15:35 plugins.txt
-rw-r--r--  1 fyre lpadmin    64 Jan 21 15:35 secret.key
-rw-r--r--  1 fyre lpadmin     0 Jan 21 15:35 secret.key.not-so-secret
drwxr-xr-x  4 fyre lpadmin   263 Jan 21 15:36 secrets
-rw-r--r--  1 fyre lpadmin     0 Jan 21 15:35 secret.yaml
drwxr-xr-x  2 fyre lpadmin    67 Jan 21 15:36 updates
drwxr-xr-x  2 fyre lpadmin    24 Jan 21 15:36 userContent
drwxr-xr-x  3 fyre lpadmin    19 Jan 21 15:35 users
drwxr-xr-x 11 fyre lpadmin  4096 Jan 21 15:35 war
drwxr-xr-x  2 fyre lpadmin     6 Jan 21 15:36 workflow-libs

with happy containers: -

docker logs 7f347b0a46e2 -f

Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: no objects passed to create
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...
Downloading plugin: credentials-binding from https://updates.jenkins.io/download/plugins/credentials-binding/1.16/credentials-binding.hpi
 > credentials-binding depends on workflow-step-api:2.10,credentials:2.1.7,plain-credentials:1.3,ssh-credentials:1.11,structs:1.7
Downloading plugin: workflow-step-api from https://updates.jenkins.io/download/plugins/workflow-step-api/latest/workflow-step-api.hpi
Downloading plugin: credentials from https://updates.jenkins.io/download/plugins/credentials/latest/credentials.hpi
Downloading plugin: plain-credentials from https://updates.jenkins.io/download/plugins/plain-credentials/latest/plain-credentials.hpi
Downloading plugin: ssh-credentials from https://updates.jenkins.io/download/plugins/ssh-credentials/latest/ssh-credentials.hpi
Downloading plugin: structs from https://updates.jenkins.io/download/plugins/structs/latest/structs.hpi
 > credentials depends on structs:1.7
 > workflow-step-api depends on structs:1.5
 > plain-credentials depends on credentials:2.1.16
 > ssh-credentials depends on credentials:2.1.17

WAR bundled plugins:


Installed plugins:
credentials-binding:1.16
credentials:2.1.18
plain-credentials:1.5
ssh-credentials:1.14
structs:1.17
workflow-step-api:2.18
Cleaning up locks
root@dmhicp-worker-3:~# 
root@dmhicp-worker-3:~# docker logs 7f347b0a46e2 -f
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
Error from server (NotFound): secrets "microclimate-ibm-microclimate" not found
error: no objects passed to create
Creating initial locks...
Analyzing war...
Registering preinstalled plugins...
Downloading plugins...
Downloading plugin: credentials-binding from https://updates.jenkins.io/download/plugins/credentials-binding/1.16/credentials-binding.hpi
 > credentials-binding depends on workflow-step-api:2.10,credentials:2.1.7,plain-credentials:1.3,ssh-credentials:1.11,structs:1.7
Downloading plugin: workflow-step-api from https://updates.jenkins.io/download/plugins/workflow-step-api/latest/workflow-step-api.hpi
Downloading plugin: credentials from https://updates.jenkins.io/download/plugins/credentials/latest/credentials.hpi
Downloading plugin: plain-credentials from https://updates.jenkins.io/download/plugins/plain-credentials/latest/plain-credentials.hpi
Downloading plugin: ssh-credentials from https://updates.jenkins.io/download/plugins/ssh-credentials/latest/ssh-credentials.hpi
Downloading plugin: structs from https://updates.jenkins.io/download/plugins/structs/latest/structs.hpi
 > credentials depends on structs:1.7
 > workflow-step-api depends on structs:1.5
 > plain-credentials depends on credentials:2.1.16
 > ssh-credentials depends on credentials:2.1.17

WAR bundled plugins:


Installed plugins:
credentials-binding:1.16
credentials:2.1.18
plain-credentials:1.5
ssh-credentials:1.14
structs:1.17
workflow-step-api:2.18
Cleaning up locks

and all of the Microclimate pods started playing nicely: -

kubectl get pods -n micro-climate

NAME                                                    READY     STATUS    RESTARTS   AGE
microclimate-ibm-microclimate-67cfd99c7b-66jj4          1/1       Running   0          29m
microclimate-ibm-microclimate-atrium-7f75d754fd-txn5p   1/1       Running   0          29m
microclimate-ibm-microclimate-devops-568c4c5989-prcbr   1/1       Running   0          29m
microclimate-jenkins-678584959-9hqld                    1/1       Running   0          29m

and a working Microclimate environment .....

No comments:

Following my previous post: - Security Bulletin: IBM Cloud Kubernetes Service is affected by a privilege escalation vulnerability in runc ...