IBM Cloud Automation Manager and the Case of the Missing Software

As per previous posts: -

Docker Secrets - And there's more ....

What do YAML and COBOL have in common ?

Another first ( for me ) - using IBM Cloud Automation Manager (CAM) to spin up stuff on IBM Cloud

I am on another voyage of discovery, this time using IBM Cloud Automation Manager (CAM) to build out IBM middleware such as DB2, WebSphere Application Server and WebSphere Liberty Profile on VMs, hosted on the IBM Cloud ( aka SoftLayer ).

So, having got CAM up-and-running, and having learned a few lessons about building VMs ( the subject of the next few posts ), I started by attempting to build out a DB2 VM ....

This went smoothly, until I hit this exception: -

...
Error: Error applying plan:
1 error(s) occurred:
* camc_softwaredeploy.DB2Node01_db2_v111_install: 1 error(s) occurred:
* camc_softwaredeploy.DB2Node01_db2_v111_install: 
**********
Error: Response from pattern manager:
StatusCode:500
Message:
{
  "failures_found": [
    "10.143.162.16     Error executing action `run` on resource 'ruby_block[fixpack_packages_validation]'", 
    "10.143.162.16     RuntimeError", 
    "10.143.162.16     116:       rescue OpenSSL::SSL::SSLError", 
    "10.143.162.16 Chef Client failed. 0 resources updated in 02 seconds", 
    "10.143.162.16 [2019-01-09T00:11:54-06:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out", 
    "10.143.162.16 [2019-01-09T00:11:54-06:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report", 
    "10.143.162.16 [2019-01-09T00:11:54-06:00] FATAL: RuntimeError: ruby_block[fixpack_packages_validation] (db2::prereq_check line 99) had an error: RuntimeError: 404 Please make sure v11.1.2fp2_linuxx64_server_t.tar.gz is available in your binary repository"
  ], 
...

It took me but a little while to work out what I'd missed ...

Having built CAM, and the underlying IBM Cloud Private (ICP) platform, I somehow assumed that the requisite IBM software packages would magically download themselves from Passport Advantage, Fix Central etc.

Well, guess what ....

They did not :-)

Using this as inspiration: -

Managing a software repository

Populating the software repository

I downloaded the required DB2 binary, fix pack and activation kit: -

ls -al /tmp/v11.1.2fp2_linuxx64_server_t.tar.gz

-rw-r--r-- 1 root root 2092136927 Jun 15  2017 /tmp/v11.1.2fp2_linuxx64_server_t.tar.gz

ls -al /tmp/DB2_Svr_11.1_Linux_x86-64.tar.gz 

-rw-r--r-- 1 root root 1945576016 Jun 11  2016 /tmp/DB2_Svr_11.1_Linux_x86-64.tar.gz

ls -al /tmp/DB2_ESE_AUSI_Activation_11.1.zip 

-rw-r--r-- 1 root root 3993254 Jun  3  2016 /tmp/DB2_ESE_AUSI_Activation_11.1.zip

and placed the files in the requisite locations: -

cp /tmp/DB2_Svr_11.1_Linux_x86-64.tar.gz /opt/ibm/docker/software-repo/var/swRepo/private/db2/v111/base

cp /tmp/v11.1.2fp2_linuxx64_server_t.tar.gz /opt/ibm/docker/software-repo/var/swRepo/private/db2/v111/maint

unzip -j -p DB2_ESE_AUSI_Activation_11.1.zip \*db2ese_u.lic > /opt/ibm/docker/software-repo/var/swRepo/private/db2/v111/license

and validated the same: -

ls -R -al /opt/ibm/docker/software-repo/var/swRepo/private/db2/v111/

/opt/ibm/docker/software-repo/var/swRepo/private/db2/v111/:

total 20

drwxr-xr-x 4 root root 4096 Jan  9 20:55 .

drwxr-xr-x 4 root root 4096 Jan  8 19:30 ..

drwxr-xr-x 2 root root 4096 Jan  9 11:45 base

-rw-r--r-- 1 root root  913 Jan  9 20:54 license

drwxr-xr-x 2 root root 4096 Jan  9 14:51 maint

/opt/ibm/docker/software-repo/var/swRepo/private/db2/v111/base:

total 1899992

drwxr-xr-x 2 root root       4096 Jan  9 11:45 .

drwxr-xr-x 4 root root       4096 Jan  9 20:55 ..

-rw-r--r-- 1 root root 1945576016 Jan  9 11:46 DB2_Svr_11.1_Linux_x86-64.tar.gz

/opt/ibm/docker/software-repo/var/swRepo/private/db2/v111/maint:

total 2043116

drwxr-xr-x 2 root root       4096 Jan  9 14:51 .

drwxr-xr-x 4 root root       4096 Jan  9 20:55 ..

-rw-r--r-- 1 root root 2092136927 Jan  9 14:51 v11.1.2fp2_linuxx64_server_t.tar.gz

With the right files in the right place, guess what, the CAM build of a DB2 VM just flippin' worked :-)

Another first ( for me ) - using IBM Cloud Automation Manager (CAM) to spin up stuff on IBM Cloud

I started playing with Terraform a few weeks back, and was able to spin up a VM on the IBM Cloud ( fka SoftLayer ) once I worked out certain parameters such as: -

• IBM API Username
• Account API Key ( Authentication Key )
• IBM Cloud API Key

via the IBM Cloud consoles: -

https://console.bluemix.net/iam/#/apikeys

https://control.softlayer.com/

https://control.softlayer.com/account/user/profile

etc.

I've now gone one step further, and am starting to spin up workloads, using IBM Cloud Automation Manager (CAM), on the self-same IBM Cloud platform.

In part, CAM requires one to have two key things: -

• A connection to the IBM Cloud - that bit was easy, as I already had the username and keys
• An Advanced Content Runtimes

That latter bit was more difficult - for me, at least :-)

From the documentation: -

Provisioning and managing your Content Runtime infrastructure

...

The Content Runtime is a virtual machine. A Chef server is installed on the virtual machine as part of the configuration of the Content Runtime. Two docker containers are installed on the machine to help communicate with the Chef server and to manage the software repository.

...

It took me a while to work out the parameters for this, and I may not have got it totally right ...

However, one thing that I did need to create was a TCP Subnet and a Virtual LAN (VLAN) on the IBM Cloud.

I ordered a new block of IP addresses from here: -

https://control.softlayer.com/network/subnets#addressSpace=ALL&subnetType=ALL

specifically this: -

4 Portable Private IP Addresses

which gave me a corresponding VLAN.

Having grabbed the subnet ( in the CIDR format of X.X.X.X/YY ) I also needed to give the newly created VLAN a "friendly" name. I also needed to grab the location - the IBM Cloud Data Centre - in my case, it's in Dallas, TX.

Like "proper" Terraform, CAM follows the same Plan and Apply phases, and provide useful feedback as one goes along.

I did see a few failures, including: -

data.ibm_network_vlan.private: data.ibm_network_vlan.private: No VLAN was found with the name '925'

ibm_compute_vm_instance.single-node: Error ordering virtual guest: SoftLayer_Exception_Public: The VLAN specified #XXXXX does not match the specified primary subnet's VLAN #XXXXX. Since the subnet supercedes the VLAN, the VLAN does not need to be provided. (HTTP 500)

ibm_compute_vm_instance.single-node: Error ordering virtual guest: SoftLayer_Exception_Public: The VLAN #XXXXX (925) is not a backend VLAN. (HTTP 500)

ibm_compute_vm_instance.single-node: Error ordering virtual guest: SoftLayer_Exception_Public: The subnet specified #XXXXX ( XXXXX ) does not currently have available slots. Free up space or remove the subnet specification. (HTTP 500)

Eventually, I realised that I needed to (a) ensure that my VLAN name was specified as the friendly name and (b) that the VLAN actually had some available IP addresses :-)

Once I did this, it all started working - the actual Apply took ~15-20 minutes, and I received lots of helpful emails from IBM Cloud before the infrastructure was properly provisioned.

So now I have the Content Runtime, time to play with some real workloads .....

This was also rather useful in providing context: -

Deploy a Cloud Automation Manager Content Runtime with an existing Chef server

What do YAML and COBOL have in common ?

They're both really really pedantic about columnar positioning ....

Whilst trying to create a Persistent Volume in my IBM Cloud Private (ICP) 3.1.1 cluster, using the following YAML: -

apiVersion: v1
kind: PersistentVolume
metadata:
  name: cam-mongo-pv
spec:
  capacity:
    storage: 15Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: dmhicp-boot.fyre.ibm.com
path: "/export/CAM_db"

the command: -

kubectl apply -f createCAM_PV1.yaml 

failed spectacularly with: -

error: error validating "createCAM_PV1.yaml": error validating data: [ValidationError(PersistentVolume): unknown field "path" in io.k8s.api.core.v1.PersistentVolume, ValidationError(PersistentVolume.spec.nfs): missing required field "path" in io.k8s.api.core.v1.NFSVolumeSource]; if you choose to ignore these errors, turn validation off with --validate=false

The solution was, of course, obvious .....

Yes, the path wasn't in line with the server element of the nfs section.

Bring back XML, all is forgiven !

Once I fixed my YAML: -

apiVersion: v1
kind: PersistentVolume
metadata:
  name: cam-mongo-pv
spec:
  capacity:
    storage: 15Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: dmhicp-boot.fyre.ibm.com
    path: "/export/CAM_db"

all was good :-)

kubectl apply -f createCAM_PV1.yaml 

persistentvolume/cam-mongo-pv created

Docker Secrets - And there's more ....

Hot on the heels of my last post: -

Shush, it's a secret .... 

having created my Docker secret: -

   kubectl create secret docker-registry --docker-username= --docker-password= --docker-email= -n services

as per this: -

Creating Docker Store secret

I made a slight mistake ....

For the --docker-username parameter, I used my email address - with which I can log into the Docker Hub: -

https://hub.docker.com/

*BUT* this proved to be a bad idea.

Whilst building out my IBM Cloud Automation Manager (CAM) environment, I saw this: -

kubectl get -n services pods

NAME                                        READY     STATUS              RESTARTS   AGE

cam-bpd-cds-5f57588776-jnw8c                0/1       Init:ErrImagePull   0          58s

cam-bpd-mariadb-6b98577f65-g75mj            0/1       Pending             0          58s

cam-bpd-mds-69f5d6988c-tfdrh                0/1       Init:ErrImagePull   0          58s

cam-bpd-ui-6c86d7d6f7-wm22d                 0/1       Pending             0          57s

cam-broker-7b86c6cff5-wcwd9                 0/1       Init:ErrImagePull   0          56s

cam-iaas-7884798b9-ztvmg                    0/1       Init:ErrImagePull   0          56s

cam-mongo-55c5976cf5-p5xtb                  0/1       ErrImagePull        0          55s

cam-orchestration-664f9647d8-sgxv2          0/1       Init:ErrImagePull   0          54s

cam-portal-ui-858b7dfcbd-qxd4k              0/1       Init:ErrImagePull   0          53s

cam-provider-helm-84b6fd45c6-zt8mt          0/1       Init:ErrImagePull   0          52s

cam-provider-terraform-79794ff875-82k4b     0/1       Init:ErrImagePull   0          52s

cam-proxy-5d64b478d6-68whm                  0/1       Init:ErrImagePull   0          51s

cam-service-composer-api-64ff5c747c-ctq5g   0/1       Init:ErrImagePull   0          49s

cam-service-composer-ui-54799748fb-dkpfn    0/1       Init:ErrImagePull   0          48s

cam-tenant-api-84f9bc79d-lsgvp              0/1       Pending             0          48s

cam-ui-basic-7d9ffc5858-b7qkr               0/1       Pending             0          47s

cam-ui-connections-6cdb6cf45b-wcw69         0/1       Pending             0          46s

cam-ui-instances-54f85dfbd4-xx2zd           0/1       Pending             0          45s

cam-ui-templates-57b467554f-l6nmh           0/1       Pending             0          44s

redis-74b9dc6d48-l4g5d                      0/1       ErrImagePull        0          50s

and, when I drilled into one of the failing pods via the IBM Cloud Private (ICP) web UI, I saw this: -

Failed to pull image "store/ibmcorp/icam-busybox:3.1.0.0-x86_64": rpc error: code = Unknown desc = Error response from daemon: Get https://registry-1.docker.io/v2/store/ibmcorp/icam-busybox/manifests/3.1.0.0-x86_64: unauthorized: incorrect username or password

I validated the exception via the CLI: -

docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.

Username: david_hay@uk.ibm.com

Password: 

Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password

I then re-read the documentation AND the prompt from docker login again, and realised that I was using my email address RATHER than my username.

Once I used my username: -

docker login

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to https://hub.docker.com to create one.

Username: davidhay1969

Password: 

Login Succeeded

Back in the ICP world, I deleted my secret: -

kubectl delete secret david-hay

and recreated it using the username rather than the email address.

I then killed the failing pod deployments: -

kubectl delete pod cam-broker-7b86c6cff5-wcwd9

kubectl delete pod cam-bpd-cds-5f57588776-jnw8c

kubectl delete pod cam-bpd-mariadb-6b98577f65-g75mj

kubectl delete pod cam-bpd-mds-69f5d6988c-tfdrh

kubectl delete pod cam-bpd-ui-6c86d7d6f7-wm22d

kubectl delete pod cam-iaas-7884798b9-ztvmg

kubectl delete pod cam-log-rotation-1546259700-zgx5v

kubectl delete pod cam-mongo-55c5976cf5-p5xtb

kubectl delete pod cam-orchestration-664f9647d8-sgxv2

kubectl delete pod cam-portal-ui-858b7dfcbd-qxd4k

kubectl delete pod cam-provider-helm-84b6fd45c6-zt8mt

kubectl delete pod cam-provider-terraform-79794ff875-82k4b

kubectl delete pod cam-proxy-5d64b478d6-68whm

kubectl delete pod cam-service-composer-api-64ff5c747c-ctq5g

kubectl delete pod cam-service-composer-ui-54799748fb-dkpfn

kubectl delete pod cam-tenant-api-84f9bc79d-lsgvp

kubectl delete pod cam-ui-basic-7d9ffc5858-b7qkr

kubectl delete pod cam-ui-connections-6cdb6cf45b-wcw69

kubectl delete pod cam-ui-instances-54f85dfbd4-xx2zd

kubectl delete pod cam-ui-templates-57b467554f-l6nmh

which then "forced" the ICP / Kubernetes Replica Set to recreate the pods.

Now I see this: -

kubectl get -n services pods

NAME                                        READY     STATUS    RESTARTS   AGE

cam-bpd-cds-5f57588776-fzq92                1/1       Running   0          28m

cam-bpd-mariadb-6b98577f65-qj7kf            0/1       Pending   0          28m

cam-bpd-mds-69f5d6988c-z9m6q                1/1       Running   0          28m

cam-bpd-ui-6c86d7d6f7-kxdrc                 0/1       Pending   0          28m

cam-broker-7b86c6cff5-kxz9r                 0/1       Pending   0          31m

cam-iaas-7884798b9-bcnhd                    1/1       Running   0          28m

cam-log-rotation-1546263600-w62hj           0/1       Pending   0          16m

cam-mongo-55c5976cf5-q5b9s                  1/1       Running   0          28m

cam-orchestration-664f9647d8-r7qxv          1/1       Running   0          28m

cam-portal-ui-858b7dfcbd-wwshz              1/1       Running   0          27m

cam-provider-helm-84b6fd45c6-rb5cl          1/1       Running   0          27m

cam-provider-terraform-79794ff875-6hh64     0/1       Pending   0          27m

cam-proxy-5d64b478d6-kwpg5                  1/1       Running   0          27m

cam-service-composer-api-64ff5c747c-2d6v9   1/1       Running   0          27m

cam-service-composer-ui-54799748fb-424lg    1/1       Running   0          27m

cam-tenant-api-84f9bc79d-bpnf5              1/1       Running   0          26m

cam-ui-basic-7d9ffc5858-drrh8               0/1       Pending   0          26m

cam-ui-connections-6cdb6cf45b-465tk         0/1       Pending   0          26m

cam-ui-instances-54f85dfbd4-r7kfv           1/1       Running   0          25m

cam-ui-templates-57b467554f-wfmpc           0/1       Pending   0          25m

redis-74b9dc6d48-l4g5d                      1/1       Running   0          1h

which is much much better.

Every day, it's a school day !

Shush, it's a secret ....

Fiddling about with IBM Cloud Private (ICP) and IBM Cloud Automation Manager (CAM), one of the pre-requisites required me to "cache" my Docker Store credentials in a Kubernetes (K8S) secrets: _

Creating Docker Store secret

The syntax is thus: -

   kubectl create secret docker-registry --docker-username= --docker-password= --docker-email= -n services

So off I went ....

The first hurdle was that my Docker password has special characters, including an ampersand ( & ), which broke the kubectl command; shells tend NOT to like ampersands in commands :-)

That was easily resolved - I just wrapped my password in double quotes ( " ), which resolved THAT particular issue.

I was using a randomly generated secret name, for no particular reason: -

DitYPtiansUP

I then hit this: -

The Secret "DitYPtiansUP" is invalid: metadata.name: Invalid value: "DitYPtiansUP": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

which didn't really help ....

I dug into the K8S documentation: -

Distribute Credentials Securely Using Secrets

and then looked at the existing secrets on my ICP cluster: -

kubectl get secrets

NAME                  TYPE                                  DATA      AGE

default-token-rvscx   kubernetes.io/service-account-token   3         2d

infra-registry-key    kubernetes.io/dockerconfigjson        1         2d

which gave me a clue ...

It looks like the secret name needs to be formatted thusly: -

• lower-case
• separated with a hyphen ( - ) or full stop / period ( . )

Therefore, I went for the path of least resistance, and used my name as my secret: -

david-hay

which did the job.

One other thing ....

This: -

kubectl get secrets

NAME                  TYPE                                  DATA      AGE

default-token-rvscx   kubernetes.io/service-account-token   3         2d

infra-registry-key    kubernetes.io/dockerconfigjson        1         2d

didn't show up my newly created secret, even though I knew it was there; I tried to create it again, and saw this: -

Error from server (AlreadyExists): secrets "david-hay" already exists

Thankfully, I realised where I was going wrong - it's all in the namespace ....

My newly created secret was placed in the services namespace, so I needed to look specifically at that: -

kubectl get secrets -n services

NAME                  TYPE                                  DATA      AGE

david-hay             kubernetes.io/dockerconfigjson        1         11m

default-token-nnz4v   kubernetes.io/service-account-token   3         2d

oauth-client-secret   Opaque                                2         2d

For the record, here's how to find the namespaces: -

kubectl get namespaces

NAME           STATUS    AGE

cert-manager   Active    2d

default        Active    2d

ibmcom         Active    2d

istio-system   Active    2d

kube-public    Active    2d

kube-system    Active    2d

platform       Active    2d

services       Active    2d

I could've done this: -

kubectl get secrets --all-namespaces=true

...

NAMESPACE      NAME                                                        TYPE                                  DATA      AGE

cert-manager   default-token-rvscx                                         kubernetes.io/service-account-token   3         2d

cert-manager   infra-registry-key                                          kubernetes.io/dockerconfigjson        1         2d

default        default-token-kj5xp                                         kubernetes.io/service-account-token   3         2d

ibmcom         default-token-5vhkl                                         kubernetes.io/service-account-token   3         2d

ibmcom         infra-registry-key                                          kubernetes.io/dockerconfigjson        1         2d

ibmcom         sa-ibmcom                                                   kubernetes.io/dockerconfigjson        1         2d

...

services       david-hay                                                   kubernetes.io/dockerconfigjson        1         16m

services       default-token-nnz4v                                         kubernetes.io/service-account-token   3         2d

services       oauth-client-secret                                         Opaque                                2         2d

...

CWOAU0062E and IBM Client Private authentication

Hmmm, I saw this: -

CWOAU0062E: The OAuth service provider could not redirect the request because the redirect URI was not valid. Contact your system administrator to resolve the problem.

whilst trying to log into my IBM Cloud Private (ICP) 3.1.1 cluster, using it's host/service name: -

https://dmhayicp-boot.fyre.ibm.com:8443

whereas it works OK using the IP address: -

https://9.20.194.53:8443

Using this: -

Accessing your IBM® Cloud Private cluster by using the management console

as reference, the suggestion is that one always use the IP address ( of the boot/management node ), but I wondered if/whether I can use the host/service name.

Reading up, I *wonder* whether I had neglected to set this: -

cluster_access_ip

in the config.yaml when I first built the cluster.

However, that appears to have "gone away" with 3.1.1, in that it's not mentioned in the KC here: -

Customizing the cluster with the config.yaml file

although it may have been replaced by: -

cluster_lb_address

...

In an environment that has multiple network interfaces (NICs), use cluster_lb_address to set a public or external IP address for the management services in your cluster. You can specify a fully-qualified domain name instead of the IP address.

This public address is assigned to the master node, used to access the console, and also used to configure kubectl.

In an HA environment, cluster_lb_address masks the cluster_vip as the leading master IP.

...

Ah well, some more digging .....


*UPDATE*

Darn, I'm good ....

I updated my config.yaml file, adding: -

cluster_lb_address: dmhayicp-boot.fyre.ibm.com

and dropped the cluster: -

docker run --net=host -t -e LICENSE=accept -v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.1.1-ee uninstall

and rebuilt the same: -

docker run --net=host -t -e LICENSE=accept -v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.1.1-ee install

and now we are in like the veritable Flynn.

I can access my cluster via this URL: -

https://dmhayicp-boot.fyre.ibm.com:8443/console/

as well as via the IP address: -

https://9.20.194.53:8443/console/

Obviously this is somewhat disruptive AND destructive.

Therefore, please please please use caution before trying this in YOUR environment.

Remember, folks, YMMV and Caveat Emptor :-)

DB2 on Windows - More fun with uninstallation

Following my earlier post: -

Fun and Games with IBM DB2 on Windows

using a different Windows 2012 R2 box, which is NOT and has NEVER been an Active Directory domain controller, I was again trying and failing to uninstall DB2.

No matter what I tried, including: -

msiexec.exe /i {5F3AC8C5-2EB8-4443-AC5D-D4AA4BD5BC21} /qb /norestart REMOVE=ALL 

where I used the previous suggestion: -

wmic product get /format:csv > c:\temp\software.csv

to get the registry ID of the DB2 installation, or this: -

db2unins.bat -f

I couldn't remove DB2 - instead, the server just logged out, leaving DB2 intact.

I did some digging into the syntax of the latter command, and found the -t tracing flag.

This time around, I did this: -

db2unins.bat -f -t c:\temp\foobar.trc

which resulted in this: -

Start to uninstall DB2 products......

Successfully killed all running DB2 processes.

Successfully removed DB2 environment variables.

Successfully cleaned the registry.

Finished manually uninstalling all the DB2 products.  Restart your computer.

and it worked !

One reboot later, back in the game ....

Easy when you know how - although rm -Rf is SO much easier !