IBM Cloud Private - Helm via the GUI - Not playing nicely

I had an interesting glitch with Helm on IBM Cloud Private earlier.

For some reason, the Helm UI, accessible via the ICP Console: -

https://192.168.1.100:8443/console/

seemed to get out-of-sync with reality, in terms of Helm repositories.

Whilst I could see multiple repositories via the CLI: -

helm repo list

NAME       URL                                                             
stable     https://kubernetes-charts.storage.googleapis.com                
local      http://127.0.0.1:8879/charts                                    
ibm-charts https://raw.githubusercontent.com/IBM/charts/master/repo/stable/

when I went into the Catalog ( Catalog -> Helm Charts ) there was nowt there: -

and the repositories didn't show up via  Manage -> Helm Repositories  : -

I also saw this: -

when I tried to add a new repository.

However, I was able to add a repo using the Helm command line: -

helm repo add ibm-charts https://raw.githubusercontent.com/IBM/charts/master/repo/stable/

I found a solution …..

Assuming that Helm was running as one of a small number of Docker containers, I checked ( on the Master/Boot node ): -

docker ps -a|grep -i helm|grep Up

d7ba0075a9a3        2cb2b0c0ca02                      "npm start"              5 hours ago         Up 5 hours                                      k8s_helmrepo_helmrepo-77dccffb66-9xwgd_kube-system_71bdb2d6-1bcb-11e8-ab0b-000c290f4d7f_26
25a17810ee7b        b72c1d4155b8                      "npm start"              5 hours ago         Up 19 minutes                                   k8s_helmapi_helm-api-5874f9d746-9qcjg_kube-system_7122aa29-1bcb-11e8-ab0b-000c290f4d7f_25
10647a5b7925        ibmcom/pause:3.0                  "/pause"                 5 hours ago         Up 5 hours                                      k8s_POD_helmrepo-77dccffb66-9xwgd_kube-system_71bdb2d6-1bcb-11e8-ab0b-000c290f4d7f_6
e47f5153f715        ibmcom/pause:3.0                  "/pause"                 5 hours ago         Up 5 hours                                      k8s_POD_helm-api-5874f9d746-9qcjg_kube-system_7122aa29-1bcb-11e8-ab0b-000c290f4d7f_6

and chose to restart the helm-api container that was actually running ( in status nom_start rather than /pause ): -

docker restart 25a17810ee7b

and monitored the logs: -

docker logs 25a17810ee7b -f

until I started seeing messages such as this: -

2018-03-01T17:11:01.426Z 'FINE' 'GET /healthcheck'
2018-03-01T17:11:01.428Z 'FINE' 'dbHealthcheck \nrepoName: ibm-charts\n'
2018-03-01T17:11:01.440Z 'FINE' 'getMessage ["statusCode",200] en '
2018-03-01T17:11:01.440Z 'FINE' 'loadMessages en'
GET /healthcheck 200 15.368 ms - 16
2018-03-01T17:11:10.117Z 'FINE' 'GET /healthcheck'
2018-03-01T17:11:10.118Z 'FINE' 'getMessage ["statusCode",200] en '
2018-03-01T17:11:10.118Z 'FINE' 'loadMessages en'
GET /healthcheck 200 0.902 ms - 16
2018-03-01T17:11:20.117Z 'FINE' 'GET /healthcheck'
2018-03-01T17:11:20.120Z 'FINE' 'getMessage ["statusCode",200] en '
2018-03-01T17:11:20.122Z 'FINE' 'loadMessages en'
GET /healthcheck 200 5.209 ms - 16

At that point, the repositories synced back up: -

and the Catalog caught up: -

So that's all good then.

The moral of the story ?

IBM Cloud Private is a container orchestration/management solution BUILT UPON CONTAINERS !

IBM Cloud Private - The credentials, they are a-changin'

This confused me briefly today :-)

I was trying to log in to an existing IBM Cloud Private (ICP) Kubernetes environment, using kubectl.

I'd previously noted down the credentials that I used previously: -

kubectl config set-cluster mycluster.icp --server=https://192.168.1.100:8001 --insecure-skip-tls-verify=true

kubectl config set-context mycluster.icp-context --cluster=mycluster.icp

kubectl config set-credentials admin --token=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImF0X2hhc2giOiI5T2FuZkVHR1Blb1g2bkI3RGRTVmxRIiwiaXNzIjoiaHR0cHM6Ly9teWNsdXN0ZXIuaWNwOjk0NDMvb2lkYy9lbmRwb2ludC9PUCIsImF1ZCI6ImM5MmYwOTZjNGNjYTVkYzM0NWIyODk2ZjI3NmNmZjdjIiwiZXhwIjoxNTE5ODUyNzI4LCJpYXQiOjE1MTk4MDk1Mjh9.K_KswVKnEeHAwNM9rFkrKm1fghP8X7jVIrOUeHbiT9YJ1RBOkDhvDgyHvbA-1sKB287W50v1ViaSMk-hty-7vA5TH_eHmWdNeVFCdJ13jN-Xq9bOn9SWYh5Q2mLsecVSwLGN-mVKDmiz-wkeI05nATlLjQB7l1RS8dl7myWpBOyRN6aXdUPmHHXzUJfWUWU9tuLGu6L5cV28Cj7FTV2ZQDqonss4XK1eDVmRIIV7EnORocaVExtuqR74wuIB3JxPyjURr0at2Hayh8-YqZYzOoDLawzPe9igPJfIJ_KAAGJYP8X71LbTCUEOqUWpxT5mB77kdCAuyAub0J1Th5alNw

kubectl config set-context mycluster.icp-context --user=admin --namespace=default

kubectl config use-context mycluster.icp-context

Now the only thing that had changed was the day/date/time, and also the fact that I'd rebooted the VMs that host the ICP Boot/Worker/Proxy nodes.

However, when I used the same set of credentials, and tried to use another kubectl command: -

kubectl get pods

I saw this: -

error: You must be logged in to the server (Unauthorized)

Now I'd previously obtained the credentials from the ICP GUI: -

https://192.168.1.100:8443/console/

However, when I grabbed them afresh today: -

kubectl config set-cluster mycluster.icp --server=https://192.168.1.100:8001 --insecure-skip-tls-verify=true

kubectl config set-context mycluster.icp-context --cluster=mycluster.icp

kubectl config set-credentials admin --token=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImF0X2hhc2giOiI1ZHkwWnJxYXE3dHNPSTBRTEtrcGhnIiwiaXNzIjoiaHR0cHM6Ly9teWNsdXN0ZXIuaWNwOjk0NDMvb2lkYy9lbmRwb2ludC9PUCIsImF1ZCI6ImM5MmYwOTZjNGNjYTVkYzM0NWIyODk2ZjI3NmNmZjdjIiwiZXhwIjoxNTE5OTUzNTQ2LCJpYXQiOjE1MTk5MTAzNDZ9.Eh0gTus1jYSWiIQsgxrWFhiAov19Su91V6uPVjsDbPCTrYC0aCgEXRSpUunSLRrOxoYU8jkAS87FtEt93UrH3gy91j-YGYGC1o2UYVXqtiyTgDzcHvCybzCpnqRZn8pH5Z9yF1G_zwE5_tRMcrrVxcGaBaOr-K7i-4RDAChs-2SdvwoehANJpe9xANdrYQR2LuZZAU4XGTDYgfmUB5E8w0QKTvNKqBU7LkLUA-hvDY8mP4UKXzKsjno4If_0XZB6eUB-iCiD4yEVOpCzzm68-gyUJ5pNOWpQmhWQ2_Ptb5rGx_4h68huRp1M1aTl8kkgmfp3gB-kdscbiscclNTZRw

kubectl config set-context mycluster.icp-context --user=admin --namespace=default

kubectl config use-context mycluster.icp-context

I noticed that the tokenised password had changed: -

Yesterday

kubectl config set-credentials admin --token=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImF0X2hhc2giOiI5T2FuZkVHR1Blb1g2bkI3RGRTVmxRIiwiaXNzIjoiaHR0cHM6Ly9teWNsdXN0ZXIuaWNwOjk0NDMvb2lkYy9lbmRwb2ludC9PUCIsImF1ZCI6ImM5MmYwOTZjNGNjYTVkYzM0NWIyODk2ZjI3NmNmZjdjIiwiZXhwIjoxNTE5ODUyNzI4LCJpYXQiOjE1MTk4MDk1Mjh9.K_KswVKnEeHAwNM9rFkrKm1fghP8X7jVIrOUeHbiT9YJ1RBOkDhvDgyHvbA-1sKB287W50v1ViaSMk-hty-7vA5TH_eHmWdNeVFCdJ13jN-Xq9bOn9SWYh5Q2mLsecVSwLGN-mVKDmiz-wkeI05nATlLjQB7l1RS8dl7myWpBOyRN6aXdUPmHHXzUJfWUWU9tuLGu6L5cV28Cj7FTV2ZQDqonss4XK1eDVmRIIV7EnORocaVExtuqR74wuIB3JxPyjURr0at2Hayh8-YqZYzOoDLawzPe9igPJfIJ_KAAGJYP8X71LbTCUEOqUWpxT5mB77kdCAuyAub0J1Th5alNw

Today

kubectl config set-credentials admin --token=eyJhbGciOiJSUzI1NiJ9.eyJzdWIiOiJhZG1pbiIsImF0X2hhc2giOiI1ZHkwWnJxYXE3dHNPSTBRTEtrcGhnIiwiaXNzIjoiaHR0cHM6Ly9teWNsdXN0ZXIuaWNwOjk0NDMvb2lkYy9lbmRwb2ludC9PUCIsImF1ZCI6ImM5MmYwOTZjNGNjYTVkYzM0NWIyODk2ZjI3NmNmZjdjIiwiZXhwIjoxNTE5OTUzNTQ2LCJpYXQiOjE1MTk5MTAzNDZ9.Eh0gTus1jYSWiIQsgxrWFhiAov19Su91V6uPVjsDbPCTrYC0aCgEXRSpUunSLRrOxoYU8jkAS87FtEt93UrH3gy91j-YGYGC1o2UYVXqtiyTgDzcHvCybzCpnqRZn8pH5Z9yF1G_zwE5_tRMcrrVxcGaBaOr-K7i-4RDAChs-2SdvwoehANJpe9xANdrYQR2LuZZAU4XGTDYgfmUB5E8w0QKTvNKqBU7LkLUA-hvDY8mP4UKXzKsjno4If_0XZB6eUB-iCiD4yEVOpCzzm68-gyUJ5pNOWpQmhWQ2_Ptb5rGx_4h68huRp1M1aTl8kkgmfp3gB-kdscbiscclNTZRw

Once I used today's set, things just worked :-)

kubectl get pods

No resources found.

kubectl get services

NAME                                          TYPE           CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
davehaydp-ibm-datapower-dev-699677858-mskvb   LoadBalancer   10.0.0.59    <pending>     9090:31059/TCP   23h
kubernetes                                    ClusterIP      10.0.0.1     <none>        443/TCP          1d

etc.

Nice :-)

VMware Workstation - Sharing my Success - With Sharing

I've been tinkering ( there's that word again ) with VMware, seeking to enable my Mac ( which is running VMware Fusion 10 ) to access VMs hosted on VMware Workstation 14.

This allows me to use Fusion to start/stop VMs that are actually hosted on an Ubuntu box ( Beast ).

In the past, I'd been tunnelling the VMware UI back from Beast to the Mac, using X11 over SSH: -

ssh -Y hayd@beast

and then: -

vmrun start /home/hayd/vmware/ICPProxy/ICPProxy.vmx
vmrun start /home/hayd/vmware/ICPWorker/ICPWorker.vmx
vmrun start /home/hayd/vmware/ICPBoot/ICPBoot.vmx
vmrun list

which means that, whilst I can start/stop the VMs from the Mac, the actual VM consoles are being "transmitted" back to the Mac via an open SSH tunnel.

That's absolutely fine ….

… until I close the lid on the Mac, at which point the tunnel drops and all of the VMs die :-(

This is a better way ….

To achieve this, I had to ensure that Sharing was enabled within VMware Workstation itself, via Edit -> Preferences : -

Note that the default port is 443 and the default location for shared VMs is /var/lib/vmware/Shared VMs 

Also, note that, as I'm running the vmware binary as a non-root user, I can't change the port.

Which is OK.

Having validated that the configuration was A-OK, I also needed to actually move my VMs from the default location of ~/vmware to the new location of /var/lib/vmware/Shared VMs

Initially, I tried doing this via a combination of mv and chmod, but to no avail.

I then realised that Workstation actually facilitates this: -

Which did the job nicely.

This is where the VMs now live ( on Beast ) : -

ls -al /var/lib/vmware/Shared\ VMs

total 28
drwxrwxrwt 7 root root 4096 Mar  1 11:46 .
drwxr-xr-x 3 root root 4096 Jan 18 09:08 ..
drwxr--r-- 3 hayd hayd 4096 Mar  1 11:46 IBM BPM v8.6
drwxrwxrwx 2 hayd hayd 4096 Mar  1 11:43 ICPBoot
drwxrwxrwx 2 hayd hayd 4096 Mar  1 11:43 ICPProxy
drwxrwxrwx 2 hayd hayd 4096 Mar  1 11:44 ICPWorker
drwxr-xr-x 2 root root 4096 Mar  1 11:31 Microsoft Windows Server 2016 (64-bit)

Having done all of that, I then used the Connect to Server wizard in Fusion: -

to connect to Beast: -

which then showed me my VMs: -

which then allowed me to power on the VM: -

For the record, on Ubuntu, VMware runs as a set of services: -

service --status-all|grep vmware

 [ - ]  vmware
 [ - ]  vmware-USBArbitrator
 [ + ]  vmware-workstation-server

service vmware status

● vmware.service - LSB: This service starts and stops VMware services
   Loaded: loaded (/etc/init.d/vmware; bad; vendor preset: enabled)
   Active: active (running) since Thu 2018-03-01 07:45:00 GMT; 4h 20min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 1891 ExecStart=/etc/init.d/vmware start (code=exited, status=0/SUCCESS)
    Tasks: 10
   Memory: 25.2M
      CPU: 569ms
   CGroup: /system.slice/vmware.service
           ├─2122 /usr/lib/vmware/bin/vmware-vmblock-fuse -o subtype=vmware-vmblock,default_permissions,allow_other /var/run/vmblock-fuse
           ├─2303 /usr/bin/vmnet-bridge -s 6 -d /var/run/vmnet-bridge-0.pid -n 0 -ienp3s0
           ├─2384 /usr/bin/vmnet-netifup -s 6 -d /var/run/vmnet-netifup-vmnet1.pid /dev/vmnet1 vmnet1
           ├─2419 /usr/bin/vmnet-dhcpd -s 6 -cf /etc/vmware/vmnet1/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet1/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet1.pid vmnet1
           ├─2476 /usr/bin/vmnet-natd -s 6 -m /etc/vmware/vmnet8/nat.mac -c /etc/vmware/vmnet8/nat/nat.conf
           ├─2603 /usr/bin/vmnet-netifup -s 6 -d /var/run/vmnet-netifup-vmnet8.pid /dev/vmnet8 vmnet8
           ├─2619 /usr/bin/vmnet-dhcpd -s 6 -cf /etc/vmware/vmnet8/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet8/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet8.pid vmnet8
           └─2668 /usr/sbin/vmware-authdlauncher

Mar 01 11:58:54 beast vmauthd[10174]: lib/ssl: protocol list tls1.2 (openssl flags 0x17000000)
Mar 01 11:58:54 beast vmauthd[10174]: lib/ssl: cipher list !aNULL:kECDH+AESGCM:ECDH+AESGCM:RSA+AESGCM:kECDH+AES:ECDH+AES:RSA+AES
Mar 01 11:58:54 beast vmauthd[10174]: Connect from remote socket (192.168.1.11:60576).
Mar 01 11:58:54 beast vmauthd[10174]: Connect from 192.168.1.11
Mar 01 11:58:54 beast vmauthd[10174]: local verification as root
Mar 01 11:58:54 beast vmauthd[10174]: login from 192.168.1.11 as 52686c92-dee3-432f-7c40-974c6ea87f3c
Mar 01 11:58:54 beast vmauthd[10174]: received CONNECT_ARGV command: /var/lib/vmware/Shared%20VMs/ICPProxy/ICPProxy.vmx mks
Mar 01 11:58:54 beast vmauthd[10174]: Acquired lock for /var/run/vmware/authd_6d49ec1496256a353d69467da74760b8
Mar 01 11:58:54 beast vmauthd[10174]: /var/lib/vmware/Shared VMs/ICPProxy/ICPProxy.vmx: Connected to mks-fd, remote end sent pid: 10148
Mar 01 11:58:54 beast vmauthd[10174]: released lock for /var/run/vmware/authd_6d49ec1496256a353d69467da74760b8

service vmware-workstation-server status

● vmware-workstation-server.service - LSB: This services starts and stops the Workstation as a Server daemon.
   Loaded: loaded (/etc/init.d/vmware-workstation-server; bad; vendor preset: enabled)
   Active: active (running) since Thu 2018-03-01 11:18:34 GMT; 48min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 7767 ExecStop=/etc/init.d/vmware-workstation-server stop (code=exited, status=0/SUCCESS)
  Process: 7820 ExecStart=/etc/init.d/vmware-workstation-server start (code=exited, status=0/SUCCESS)
    Tasks: 63
   Memory: 22.0G
      CPU: 6min 4.288s
   CGroup: /system.slice/vmware-workstation-server.service
           ├─ 7855 /usr/lib/vmware/bin/vmware-hostd -a /etc/vmware/hostd/config.xml
           ├─10089 /usr/lib/vmware/bin/vmware-vmx -s sched.group= -# product=1;name=VMware Workstation;version=14.1.1;buildnumber=7528167;licensename=VMware Workstation;licenseversion=14.0; -@ transport
           └─10148 /usr/lib/vmware/bin/vmware-vmx -s sched.group= -# product=1;name=VMware Workstation;version=14.1.1;buildnumber=7528167;licensename=VMware Workstation;licenseversion=14.0; -@ transport

Mar 01 11:33:31 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 127.0.0.1
Mar 01 11:33:37 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 127.0.0.1
Mar 01 11:34:23 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 127.0.0.1
Mar 01 11:34:41 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 192.168.1.11
Mar 01 11:35:44 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 192.168.1.11
Mar 01 11:36:35 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 192.168.1.11
Mar 01 11:38:45 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 192.168.1.11
Mar 01 11:42:08 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 127.0.0.1
Mar 01 11:50:15 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 127.0.0.1
Mar 01 11:56:51 beast /usr/lib/vmware/bin/vmware-hostd[7855]: Accepted password for user hayd from 192.168.1.11

and there's a corresponding log for the VMware Host Daemon: -

ls -altrc /var/log/vmware/hostd.log 

lrwxrwxrwx 1 root root 29 Mar  1 11:18 /var/log/vmware/hostd.log -> /var/log/vmware//hostd-44.log

IT Security - what do I know ?

Well, I'm constantly adding to the list of things that I know ( in full or in part )

To quote Donald Rumsfeld: -

"...because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns – the ones we don't know we don't know…"

Source: Wikipedia

These are some of my sources: -

https://www.feistyduck.com/

https://www.securemac.com/topics/checklist

https://risky.biz/

https://twit.tv/shows/security-now

and these are those who I choose to follow on Twitter: -

Patrick Gray https://twitter.com/riskybusiness

Adam Boileau https://twitter.com/metlstorm?lang=en

Steve Gibson https://twitter.com/sggrc?lang=en

Ivan Ristic https://twitter.com/ivanristic

Scott Helme https://twitter.com/scott_helme?lang=en

Swift On Security https://twitter.com/SwiftOnSecurity

Duo Security https://twitter.com/duosec

etc.

IBM Cloud Private - 502 Bad Gateway

I saw this: -

after resuming three VMs that host my IBM Cloud Private (ICP) cluster.

Assuming that the VM sleep had borked things, I followed this: -

What is the proper way to shut down ICp?

to shut down and restart Docker and Kubelet: -

sudo systemctl stop kubelet

sudo systemctl stop docker

sudo systemctl start kubelet

sudo systemctl start docker

but to no avail.

I even rebooted the VMs ( having again stopped Docker and Kubelet ), but still no dice :-(

Running this: -

sudo journalctl -e -u kubelet

on the Boot node ( VM ) showed me this: -

Feb 28 09:13:54 icpboot.uk.ibm.com hyperkube[856]: W0228 09:13:54.072049     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: I0228 09:14:04.003480     856 kuberuntime_manager.go:500] Container {Name:iam-policy-decision Image:ibmcom/iam-policy-decision:2.1.0.1 Command:[] Args:
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: n:false StdinOnce:false TTY:false} is dead, but RestartPolicy says that we should restart it.
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: I0228 09:14:04.004425     856 kuberuntime_manager.go:739] checking backoff for container "iam-policy-decision" in pod "auth-pdp-bpk7h_kube-system(b5ddc
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:04.190524     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:04.193761     856 kuberuntime_container.go:191] Non-root verification doesn't support non-numeric user (iam)
Feb 28 09:14:14 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:14.313716     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:24 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:24.350819     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:34 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:34.484130     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:44 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:44.564230     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:14:54 icpboot.uk.ibm.com hyperkube[856]: W0228 09:14:54.641818     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:04 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:04.763905     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:15 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:15.138129     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:25 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:25.268208     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:35 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:35.303902     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:45 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:45.377637     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available
Feb 28 09:15:55 icpboot.uk.ibm.com hyperkube[856]: W0228 09:15:55.508744     856 helpers.go:847] eviction manager: no observation found for eviction signal allocatableNodeFs.available

However, it may have just been timing, because, after ~10 minutes, things started to improve.

I again hit the URL: -

https://192.168.1.100:8443/console/

and got a login page and, post-login, this: -

Patience is the key !

WebSphere Application Server to Oracle - Listener Says No

I saw this: -

The test connection operation failed for data source BPM Business Space data source on server nodeagent at node Node1 with the following exception: java.sql.SQLException: IO Error: The Network Adapter could not establish the connection DSRA0010E: SQL State = 08006, Error Code = 17,002. View JVM logs for further details.

whilst testing an IBM BPM database connection to Oracle 12c.

This was my WAS JDBC Datasource configuration: -

jdbc:oracle:thin:@//bpm856.uk.ibm.com:1521/orcl.uk.ibm.com

In the WAS Node Agent log, I saw this: -

[23/02/18 21:13:33:393 GMT] 00000066 DataSourceCon E   DSRA8040I: Failed to connect to the DataSource "".  Encountered java.sql.SQLException: IO Error: The Network Adapter could not establish the connection DSRA0010E: SQL State = 08006, Error Code = 17,002
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:673)
at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:715)
at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:385)
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:30)
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:564)
at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:303)
at oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(OracleXADataSource.java:500)
at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:174)
at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:143)
at com.ibm.ws.rsadapter.DSConfigHelper$1.run(DSConfigHelper.java:1267)
at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5477)
at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5603)
at com.ibm.ws.security.core.SecurityContext.runAsSystem(SecurityContext.java:255)
at com.ibm.ws.rsadapter.spi.ServerFunction$6.run(ServerFunction.java:567)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.rsadapter.DSConfigHelper.getPooledConnection(DSConfigHelper.java:1282)
at com.ibm.ws.rsadapter.DSConfigHelper.getPooledConnection(DSConfigHelper.java:1190)
at com.ibm.ws.rsadapter.DSConfigurationHelper.getConnectionFromDSOrPooledDS(DSConfigurationHelper.java:2075)
at com.ibm.ws.rsadapter.DSConfigurationHelper.getConnectionFromDSOrPooledDS(DSConfigurationHelper.java:1951)
at com.ibm.ws.rsadapter.DSConfigurationHelper.testConnectionForGUI(DSConfigurationHelper.java:2819)
at sun.reflect.GeneratedMethodAccessor39.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at com.ibm.ws.management.DataSourceConfigHelperMBean.testConnectionToDataSource2(DataSourceConfigHelperMBean.java:556)
at com.ibm.ws.management.DataSourceConfigHelperMBean.testConnection(DataSourceConfigHelperMBean.java:484)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:95)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:88)
at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:56)
at java.lang.reflect.Method.invoke(Method.java:620)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:292)
at javax.management.modelmbean.RequiredModelMBean$4.run(RequiredModelMBean.java:1261)
at java.security.AccessController.doPrivileged(AccessController.java:311)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:88)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1255)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:1093)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:832)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:814)
at com.ibm.ws.management.AdminServiceImpl$1.run(AdminServiceImpl.java:1335)
at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
at com.ibm.ws.management.AdminServiceImpl.invoke(AdminServiceImpl.java:1228)
at com.ibm.ws.management.connector.AdminServiceDelegator.invoke(AdminServiceDelegator.java:181)
at com.ibm.ws.management.connector.ipc.CallRouter.route(CallRouter.java:247)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink.doWork(IPCConnectorInboundLink.java:360)
at com.ibm.ws.management.connector.ipc.IPCConnectorInboundLink$IPCConnectorReadCallback.complete(IPCConnectorInboundLink.java:602)
at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1818)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:175)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1881)
Caused by: java.lang.Exception: The Network Adapter could not establish the connection
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:445)
at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:464)
at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:594)
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:229)
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1360)
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:486)
... 56 more
Caused by: java.net.ConnectException: Connection refused
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:370)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:231)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:213)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:404)
at java.net.Socket.connect(Socket.java:643)
at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:162)
at oracle.net.nt.ConnOption.connect(ConnOption.java:133)
at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:411)
... 61 more

From an Oracle perspective, I checked the Listener configuration: -

lsnrctl start LISTENER

  LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 23-FEB-2018 21:16:14

  Copyright (c) 1991, 2016, Oracle.  All rights reserved.

  Starting /home/oracle/app/oracle/product/12.2.0/dbhome_1/bin/tnslsnr: please wait...

  TNSLSNR for Linux: Version 12.2.0.1.0 - Production
  System parameter file is /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
  Log messages written to /home/oracle/app/oracle/diag/tnslsnr/bpm856/listener/alert/log.xml
  Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
  Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))

  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))
  STATUS of the LISTENER
  ------------------------
  Alias                     LISTENER
  Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
  Start Date                23-FEB-2018 21:16:14
  Uptime                    0 days 0 hr. 0 min. 0 sec
  Trace Level               off
  Security                  ON: Local OS Authentication
  SNMP                      OFF
  Listener Parameter File   /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
  Listener Log File         /home/oracle/app/oracle/diag/tnslsnr/bpm856/listener/alert/log.xml
  Listening Endpoints Summary...
    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
    (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  The listener supports no services
  The command completed successfully

lsnrctl status listener

  LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 23-FEB-2018 21:16:17

  Copyright (c) 1991, 2016, Oracle.  All rights reserved.

  Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))
  STATUS of the LISTENER
  ------------------------
  Alias                     LISTENER
  Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
  Start Date                23-FEB-2018 21:16:14
  Uptime                    0 days 0 hr. 0 min. 3 sec
  Trace Level               off
  Security                  ON: Local OS Authentication
  SNMP                      OFF
  Listener Parameter File   /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
  Listener Log File         /home/oracle/app/oracle/diag/tnslsnr/bpm856/listener/alert/log.xml
  Listening Endpoints Summary...
    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
    (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
  The listener supports no services
  The command completed successfully

I also validated the Service Name: -

sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Fri Feb 23 21:19:24 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>  select value from v$parameter where name='service_names';

VALUE
--------------------------------------------------------------------------------
orcl.uk.ibm.com

SQL>  exit

Yep, that was the problem - I was trying to connect to the Listener using the hostname bpm856.uk.ibm.com whereas the Listener was configured to use127.0.0.1.

Once I updated the listener: -

vi /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora

from: -

# listener.ora Network Configuration File: /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
# Generated by Oracle configuration tools.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 127.0.0.1)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

to: -

# listener.ora Network Configuration File: /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
# Generated by Oracle configuration tools.

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST = bpm856.uk.ibm.com)(PORT = 1521))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
    )
  )

and restarted the Listener: -

lsnrctl start LISTENER

lsnrctl start LISTENER

and re-validated the configuration: -

lsnrctl status listener

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 23-FEB-2018 21:33:20

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=bpm856.uk.ibm.com)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                23-FEB-2018 21:18:08
Uptime                    0 days 0 hr. 15 min. 11 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /home/oracle/app/oracle/product/12.2.0/dbhome_1/network/admin/listener.ora
Listener Log File         /home/oracle/app/oracle/diag/tnslsnr/bpm86/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=bpm86.uk.ibm.com)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1521)))
Services Summary...
Service "orcl.uk.ibm.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
Service "orclXDB.uk.ibm.com" has 1 instance(s).
  Instance "orcl", status READY, has 1 handler(s) for this service...
The command completed successfully

we were good to go :-)

Oracle 12c on Linux - Fonts Not So Good

Here we go again ….

Installing Oracle 12c on a Red Hat Enterprise Linux 7.4 box: -

/tmp/database/runInstaller

Starting Oracle Universal Installer...

Checking Temp space: must be greater than 500 MB.   Actual 13067 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 2076 MB    Passed
Checking monitor: must be configured to display at least 256 colors
    >>> Could not execute auto check for display colors using command /usr/bin/xdpyinfo. Check if the DISPLAY variable is set.    Failed <<<<

Some requirement checks failed. You must fulfill these requirements before

continuing with the installation,

Continue? (y/n) [n] y

>>> Ignoring required pre-requisite failures. Continuing...
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2018-02-23_07-10-09PM. Please wait ...[oracle@bpm86 ~]$ Fontconfig error: "local.conf", line 2: XML or text declaration not at start of entity
Exception in thread "main" java.lang.ExceptionInInitializerError
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
at javax.swing.UIDefaults.getUI(UIDefaults.java:769)
at javax.swing.UIManager.getUI(UIManager.java:1016)
at javax.swing.JComboBox.updateUI(JComboBox.java:266)
at javax.swing.JComboBox.init(JComboBox.java:231)
at javax.swing.JComboBox.<init>(JComboBox.java:183)
at oracle.help.DefaultNavigatorPanel$MinimumSizedComboBox.<init>(Unknown Source)
at oracle.help.DefaultNavigatorPanel.<init>(Unknown Source)
at oracle.help.Help._initHelpSystem(Unknown Source)
at oracle.help.Help.<init>(Unknown Source)
at oracle.help.Help.<init>(Unknown Source)
at oracle.help.Help.<init>(Unknown Source)
at oracle.install.commons.util.HelpManager.loadHelp(HelpManager.java:230)
at oracle.install.commons.util.Application.startup(Application.java:943)
at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:181)
at oracle.install.commons.flow.FlowApplication.startup(FlowApplication.java:198)
at oracle.install.commons.base.driver.common.Installer.startup(Installer.java:566)
at oracle.install.ivw.db.driver.DBInstaller.startup(DBInstaller.java:127)
at oracle.install.ivw.db.driver.DBInstaller.main(DBInstaller.java:165)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 0
at sun.font.CompositeStrike.getStrikeForSlot(CompositeStrike.java:75)
at sun.font.CompositeStrike.getFontMetrics(CompositeStrike.java:93)
at sun.font.FontDesignMetrics.initMatrixAndMetrics(FontDesignMetrics.java:359)
at sun.font.FontDesignMetrics.<init>(FontDesignMetrics.java:350)
at sun.font.FontDesignMetrics.getMetrics(FontDesignMetrics.java:302)
at sun.swing.SwingUtilities2.getFontMetrics(SwingUtilities2.java:1113)
at javax.swing.JComponent.getFontMetrics(JComponent.java:1626)
at javax.swing.text.PlainView.calculateLongestLine(PlainView.java:639)
at javax.swing.text.PlainView.updateMetrics(PlainView.java:209)
at javax.swing.text.PlainView.updateDamage(PlainView.java:527)
at javax.swing.text.PlainView.insertUpdate(PlainView.java:451)
at javax.swing.text.FieldView.insertUpdate(FieldView.java:293)
at javax.swing.plaf.basic.BasicTextUI$RootView.insertUpdate(BasicTextUI.java:1610)
at javax.swing.plaf.basic.BasicTextUI$UpdateHandler.insertUpdate(BasicTextUI.java:1869)
at javax.swing.text.AbstractDocument.fireInsertUpdate(AbstractDocument.java:201)
at javax.swing.text.AbstractDocument.handleInsertString(AbstractDocument.java:748)
at javax.swing.text.AbstractDocument.insertString(AbstractDocument.java:707)
at javax.swing.text.PlainDocument.insertString(PlainDocument.java:130)
at javax.swing.text.AbstractDocument.replace(AbstractDocument.java:669)
at javax.swing.text.JTextComponent.setText(JTextComponent.java:1669)
at javax.swing.JTextField.<init>(JTextField.java:243)
at javax.swing.JTextField.<init>(JTextField.java:183)
at com.jgoodies.looks.plastic.PlasticComboBoxUI.<clinit>(PlasticComboBoxUI.java:88)
... 27 more

As ever, Google had the answer ( Google is my friend ) ….

Oracle Database 12 install crashes on Oracle Linux 7u4

The TL;DR; was: -

su -

vi /etc/fonts/local.conf

<?xml version='1.0'?>
<!DOCTYPE fontconfig SYSTEM 'fonts.dtd'>
<fontconfig>
  <alias>
    <family>serif</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
  <alias>
    <family>sans-serif</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
  <alias>
    <family>monospace</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
  <alias>
    <family>dialog</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
  <alias>
    <family>dialoginput</family>
    <prefer><family>Utopia</family></prefer>
  </alias>
</fontconfig>

Job done :-)