Friday, 14 December 2018

Consume and provide APIs with API Connect and Node.js

From a friend: -


When organizations need to expose functionality to the outside world, they can do so at a business-to-business level through collaboration or through a front-end service for customers. In the past, they would have used Service Oriented Architecture (SOA) practices to create web services which could be reused for each new business wishing to use the same functionality.

Recently, organizations have looked to take advantage of the API economy which exposes the services through APIs. There are many key differences between services and APIs.

IBM API Connect allows organizations to manage, run, and secure the APIs they provide. Their Developer Portal exposes the APIs for discovery so consumers can subscribe to the APIs and call them with their own applications.

This tutorial will demonstrates both the API provider and API consumer journey using IBM API Connect as an API hub. We cover basic concepts and show API providers how to use Node.js for internal service and how to use Node.js in an application for the API consumer to call the API.

Thursday, 13 December 2018

SQL1022C There is not enough memory available to process the command.

So this occurred again: -

2018-12-13-20.21.11.332000+000 I64925F462           LEVEL: Event
PID     : 1476                 TID : 1488           PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
HOSTNAME: WIN2012
EDUID   : 1488
FUNCTION: DB2 UDB, base sys utilities, DB2StartMain, probe:5792
MESSAGE : ZRC=0xFFFFFC02=-1022
          SQL1022C  There is not enough memory available to process the
          command.


following a similar experience a few years back: -

As with last time, this occurred after I switched a Windows server ( this time Windows 2012 R2 ) from a standalone server to an Active Directory domain controller.

This time around, I think I fixed it more quickly ….

It's al to do with Extended Security …

Even though the actual hostname didn't change, I wonder whether that broke it …

This is relevant: -

Adding extended security after installation (db2extsec command)

If the Db2 database system was installed without extended security enabled, you can enable it by executing the command db2extsec. To execute the db2extsec command you must be a member of the local Administrators group so that you have the authority to modify the ACL of the protected objects.

You can run the db2extsec command multiple times, if necessary, however, if this is done, you cannot disable extended security unless you issue the db2extsec -r command immediately after each execution of db2extsec.

Removing extended security

CAUTION
Do not remove extended security after it has been enabled unless absolutely necessary.


You can remove extended security by running the command db2extsec -r, however, this will only succeed if no other database operations (such as creating a database, creating a new instance, adding table spaces, and so on) have been performed after enabling extended security. The safest way to remove the extended security option is to uninstall the Db2 database system, delete all the relevant Db2 directories (including the database directories) and then reinstall the Db2 database system without extended security enabled.


Note the caution - for me, this is ONLY a demo/test box so it's less of a concern.

This also helped: -


I did check the db2nodes.cfg as that's caused me no end of fun on Unix in the past: -

but both were OK: -

type "C:\ProgramData\IBM\DB2\DB2COPY1\DB2\db2nodes.cfg"

0 WIN2012 WIN2012 0

type "C:\Users\All Users\IBM\DB2\DB2COPY1\DB2\db2nodes.cfg"

0 WIN2012 WIN2012 0

which matched the hostname: -

hostname

WIN2012

In the end, I removed Extended Security, and things worked OK even after a reboot: -

db2extsec /r

Obviously I do NOT NOT NOT recommend disabling security - if this happens to you, call IBM Support.

Remember, YMMV and Caveat Emptor

IBM Cloud Private 3.1.1 - Debugging an installation

I’ve been tinkering with an ICP 3.1.1 deployment, and kept seeing the same exception at the end of the installation: -

left).
fatal: [localhost]: FAILED! => changed=true 
 attempts: 100
 cmd: kubectl -n kube-system get daemonset auth-pdp -o=custom-columns=A:.status.numberAvailable,B:.status.desiredNumberScheduled --no-headers=true | tr -s " " | awk '$1 == $2 {print "READY"}'
 delta: '0:00:01.308879'
 end: '2018-11-22 17:00:56.092611'
 rc: 0
 start: '2018-11-22 17:00:54.783732'
 stderr: ''
 stderr_lines:
 stdout: ''
 stdout_lines:

along with loads of these: -

FAILED - RETRYING: Waiting for auth-pdp to start (100 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (99 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (98 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (97 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (96 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (95 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (94 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (93 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (92 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (91 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (90 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (89 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (88 retries left).

FAILED - RETRYING: Waiting for auth-pdp to start (9 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (8 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (7 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (6 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (5 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (4 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (3 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (2 retries left).
FAILED - RETRYING: Waiting for auth-pdp to start (1 retries left).

...

I suspected that I was hitting a resource constraint, in terms of CPU or RAM.

Looking here: -



My wonder team suggested a number of debugging commands, all of which I ran from the boot/master node: -

docker ps -a | grep pdp

- This showed NOTHING running

kubectl get ds -n kube-system

kubectl describe node 9.20.194.53
kubectl describe ds auth-pdp -n kube-system
the last of which threw this up: -

FailedPlacement - Failed to place pod on 9.20.194.53: Node didn't have enough resource

which did confirm that it WAS a resource constraint.

My ICP cluster has three nodes: -
  • Boot/Master
  • Management
  • Worker
  • Proxy
as it’s just a test environment.

The Boot/Master node ONLY had 2 CPU cores and 16 GB RAM.

I dynamically increased the CPU cores from 2 to 8, which is the recommended minimum number, as per this: -


and uninstalled: -

cd /opt/ibm-cloud-private-3.1.1/cluster
docker run --net=host -t -e LICENSE=accept -v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.1.1-ee uninstall

and then reinstalled: -

docker run --net=host -t -e LICENSE=accept -v "$(pwd)":/installer/cluster ibmcom/icp-inception-amd64:3.1.1-ee install

After an hour or so, this finished A-OK, with: -

PLAY [Uploading images and charts of archive addons] ***************************

TASK [archive-addon : include_tasks] *******************************************

PLAY RECAP *********************************************************************
9.20.194.53                : ok=157  changed=95   unreachable=0    failed=0   
9.20.194.58                : ok=102  changed=57   unreachable=0    failed=0   
9.20.194.61                : ok=167  changed=107  unreachable=0    failed=0   
9.20.194.95                : ok=101  changed=56   unreachable=0    failed=0   
localhost                  : ok=248  changed=155  unreachable=0    failed=0   


POST DEPLOY MESSAGE ************************************************************

The Dashboard URL: https://9.20.194.53:8443, default username/password is admin/admin

Playbook run took 0 days, 0 hours, 54 minutes, 15 seconds
For reference, the logs are located here: -

ls -altrc /opt/ibm-cloud-private-3.1.1/cluster/logs

total 368
-rw-r--r-- 1 root root 180108 Nov 23 16:58 install.log.20181123141759
-rw-r--r-- 1 root root  18130 Nov 26 14:34 uninstall.log.20181126143338
drwxr-xr-x 3 root root    125 Nov 26 14:45 .
drwxr-xr-x 2 root root     80 Nov 26 14:46 .detail
drwxr-xr-x 9 root root    184 Nov 26 15:35 ..
-rw-r--r-- 1 root root 175228 Nov 26 15:40 install.log.20181126144552

Now that things are working, the debug commands are also looking good: -

docker ps -a | grep pdp
5474a1ae3020        5a7e7a8abb4b                          "bash -c ./startiam.…"   About an hour ago   Up About an hour                                     k8s_auth-pdp_auth-pdp-jlb6f_kube-system_fc825fe7-f18d-11e8-b6fe-00000914c235_0
85dcec4493eb        769824455743                          "audit-entrypoint.sh"    About an hour ago   Up About an hour                                     k8s_icp-audit-service_auth-pdp-jlb6f_kube-system_fc825fe7-f18d-11e8-b6fe-00000914c235_0
d79e794efbf1        493b365fcc13                          "sh -c 'until curl -…"   About an hour ago   Exited (0) About an hour ago                         k8s_init-pap_auth-pdp-jlb6f_kube-system_fc825fe7-f18d-11e8-b6fe-00000914c235_0
00e86d7fcc18        493b365fcc13                          "sh -c 'until curl -…"   About an hour ago   Exited (0) About an hour ago                         k8s_init-token-service_auth-pdp-jlb6f_kube-system_fc825fe7-f18d-11e8-b6fe-00000914c235_0
1295d90a2fca        493b365fcc13                          "sh -c 'until curl -…"   About an hour ago   Exited (0) About an hour ago                         k8s_init-identity-manager_auth-pdp-jlb6f_kube-system_fc825fe7-f18d-11e8-b6fe-00000914c235_0
7609629cd307        493b365fcc13                          "sh -c 'until curl -…"   About an hour ago   Exited (0) About an hour ago                         k8s_init-identity-provider_auth-pdp-jlb6f_kube-system_fc825fe7-f18d-11e8-b6fe-00000914c235_0
44523d1ef769        493b365fcc13                          "sh -c 'until curl -…"   About an hour ago   Exited (0) About an hour ago                         k8s_init-auth-service_auth-pdp-jlb6f_kube-system_fc825fe7-f18d-11e8-b6fe-00000914c235_0
e45b13e73bed        mycluster.icp:8500/ibmcom/pause:3.1   "/pause"                 About an hour ago   Up About an hour                                     k8s_POD_auth-pdp-jlb6f_kube-system_fc825fe7-f18d-11e8-b6fe-00000914c235_0
kubectl get ds -n kube-system

NAME                                 DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
audit-logging-fluentd-ds             4         4         4         4            4                     40m
auth-apikeys                         1         1         1         1            1           master=true     51m
auth-idp                             1         1         1         1            1           master=true     51m
auth-pap                             1         1         1         1            1           master=true     51m
auth-pdp                             1         1         1         1            1           master=true     51m
calico-node                          4         4         4         4            4                     54m
catalog-ui                           1         1         1         1            1           master=true     41m
icp-management-ingress               1         1         1         1            1           master=true     51m
kube-dns                             1         1         1         1            1           master=true     54m
logging-elk-filebeat-ds              4         4         4         4            4                     46m
metering-reader                      4         4         4         4            4                     40m
monitoring-prometheus-nodeexporter   4         4         4         4            4                     40m
nginx-ingress-controller             1         1         1         1            1           proxy=true      53m
nvidia-device-plugin                 4         4         4         4            4                     53m
platform-ui                          1         1         1         1            1           master=true     40m
service-catalog-apiserver            1         1         1         1            1           master=true     53m
unified-router                       1         1         1         1            1           master=true     40m
kubectl describe ds auth-pdp -n kube-system
Name:           auth-pdp
Selector:       component=auth-pdp,k8s-app=auth-pdp,release=auth-pdp
Node-Selector:  master=true
Labels:         app=auth-pdp
                chart=auth-pdp-3.1.1
                component=auth-pdp
                heritage=Tiller
                release=auth-pdp
Annotations:    
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:       component=auth-pdp
                k8s-app=auth-pdp
                release=auth-pdp
  Annotations:  scheduler.alpha.kubernetes.io/critical-pod=
  Init Containers:
   init-auth-service:
    Image:      mycluster.icp:8500/ibmcom/icp-platform-auth:3.1.1
    Port:       
    Host Port:  
    Command:
      sh
      -c
      until curl -k -i -fsS https://platform-auth-service:9443/oidc/endpoint/OP/.well-known/openid-configuration | grep "200 OK"; do sleep 3; done;
    Environment:  
    Mounts:       
   init-identity-provider:
    Image:      mycluster.icp:8500/ibmcom/icp-platform-auth:3.1.1
    Port:       
    Host Port:  
    Command:
      sh
      -c
      until curl --cacert /certs/ca.crt -i -fsS https://platform-identity-provider:4300 | grep "200 OK"; do sleep 3; done;
    Environment:  
    Mounts:
      /certs from cluster-ca (rw)
   init-identity-manager:
    Image:      mycluster.icp:8500/ibmcom/icp-platform-auth:3.1.1
    Port:       
    Host Port:  
    Command:
      sh
      -c
      until curl --cacert /certs/ca.crt -i -fsS https://platform-identity-management:4500 | grep "200 OK"; do sleep 3; done;
    Environment:  
    Mounts:
      /certs from cluster-ca (rw)
   init-token-service:
    Image:      mycluster.icp:8500/ibmcom/icp-platform-auth:3.1.1
    Port:       
    Host Port:  
    Command:
      sh
      -c
      until curl -k -i -fsS https://iam-token-service:10443/oidc/keys | grep "200 OK"; do sleep 3; done;
    Environment:  
    Mounts:       
   init-pap:
    Image:      mycluster.icp:8500/ibmcom/icp-platform-auth:3.1.1
    Port:       
    Host Port:  
    Command:
      sh
      -c
      until curl --cacert /certs/ca.crt -i -fsS https://iam-pap:39001/v1/health | grep "200 OK"; do sleep 3; done;
    Environment:  
    Mounts:
      /certs from cluster-ca (rw)
  Containers:
   icp-audit-service:
    Image:      mycluster.icp:8500/ibmcom/icp-audit-service:3.1.1
    Port:       
    Host Port:  
    Limits:
      cpu:     200m
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  256Mi
    Environment:
      AUDIT_DIR:  /app/logs/audit
    Mounts:
      /app/logs/audit from shared (rw)
      /etc/logrotate.conf from logrotate-conf (rw)
      /etc/logrotate.d/audit from logrotate (rw)
      /run/systemd/journal from journal (rw)
   auth-pdp:
    Image:      mycluster.icp:8500/ibmcom/iam-policy-decision:3.1.1
    Port:       
    Host Port:  
    Requests:
      cpu:      500m
      memory:   512Mi
    Readiness:  http-get http://:7998/v1/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DEFAULT_ADMIN_USER:        Optional: false
      AUDIT_ENABLED:                               Optional: false
      DEFAULT_ADMIN_PASSWORD:    Optional: false
      POD_NAME:                 (v1:metadata.name)
      POD_NAMESPACE:            (v1:metadata.namespace)
      CLUSTER_NAME:              Optional: false
      MONGO_DB:                platform-db
      MONGO_COLLECTION:        iam
      MONGO_USERNAME:                Optional: false
      MONGO_PASSWORD:            Optional: false
      MONGO_HOST:              mongodb
      MONGO_PORT:              27017
      MONGO_AUTHSOURCE:        admin
      CF_DB_NAME:              security-data
      DB_NAME:                 platform-db
      CAMS_PDP_URL:            http://iam-pdp:7998
      IAM_TOKEN_SERVICE_URL:   https://iam-token-service:10443
      IDENTITY_PROVIDER_URL:   https://platform-identity-provider:4300
      IAM_PAP_URL:             https://iam-pap:39001
      DEFAULT_TTL:               Optional: false
    Mounts:
      /app/logs/audit from shared (rw)
      /certs from cluster-ca (rw)
      /certs/mongodb-ca from mongodb-ca-cert (rw)
      /certs/mongodb-client from mongodb-client-cert (rw)
  Volumes:
   mongodb-ca-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cluster-ca-cert
    Optional:    false
   cluster-ca:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  cluster-ca-cert
    Optional:    false
   journal:
    Type:          HostPath (bare host directory volume)
    Path:          /run/systemd/journal
    HostPathType:  
   shared:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
   logrotate:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      auth-pdp
    Optional:  false
   logrotate-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      auth-pdp
    Optional:  false
   mongodb-client-cert:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  icp-mongodb-client-cert
    Optional:    false
Events:
  Type    Reason            Age   From                  Message
  ----    ------            ----  ----                  -------
  Normal  SuccessfulCreate  52m   daemonset-controller  Created pod: auth-pdp-jlb6f

kubectl describe node 9.20.194.53

Name:               9.20.194.53
Roles:              etcd,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    etcd=true
                    kubernetes.io/hostname=9.20.194.53
                    master=true
                    node-role.kubernetes.io/etcd=true
                    node-role.kubernetes.io/master=true
                    role=master
Annotations:        node.alpha.kubernetes.io/ttl=0
                    volumes.kubernetes.io/controller-managed-attach-detach=true
CreationTimestamp:  Mon, 26 Nov 2018 14:48:44 +0000
Taints:             dedicated=infra:NoSchedule
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  OutOfDisk        False   Mon, 26 Nov 2018 16:07:34 +0000   Mon, 26 Nov 2018 14:48:44 +0000   KubeletHasSufficientDisk     kubelet has sufficient disk space available
  MemoryPressure   False   Mon, 26 Nov 2018 16:07:34 +0000   Mon, 26 Nov 2018 14:48:44 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 26 Nov 2018 16:07:34 +0000   Mon, 26 Nov 2018 14:48:44 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 26 Nov 2018 16:07:34 +0000   Mon, 26 Nov 2018 14:48:44 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 26 Nov 2018 16:07:34 +0000   Mon, 26 Nov 2018 15:11:48 +0000   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  9.20.194.53
  Hostname:    9.20.194.53
Capacity:
 cpu:                8
 ephemeral-storage:  249436164Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             16424812Ki
 pods:               80
Allocatable:
 cpu:                8
 ephemeral-storage:  249333764Ki
 hugepages-1Gi:      0
 hugepages-2Mi:      0
 memory:             16322412Ki
 pods:               80
System Info:
 Machine ID:                 428e44fb1ec74efba5d4e3ca11fa2ac9
 System UUID:                9E82ABA4-CABA-4645-B285-409E35FDF986
 Boot ID:                    321104d1-77ca-48ee-af9d-8f8311a749a5
 Kernel Version:             4.15.0-38-generic
 OS Image:                   Ubuntu 18.04.1 LTS
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  docker://18.3.1
 Kubelet Version:            v1.11.3+icp-ee
 Kube-Proxy Version:         v1.11.3+icp-ee
Non-terminated Pods:         (35 in total)
  Namespace                  Name                                                  CPU Requests  CPU Limits   Memory Requests  Memory Limits
  ---------                  ----                                                  ------------  ----------   ---------------  -------------
  cert-manager               ibm-cert-manager-cert-manager-7d656f5dd5-c7lqt        0 (0%)        0 (0%)       0 (0%)           0 (0%)
  kube-system                audit-logging-fluentd-ds-lsjs2                        0 (0%)        0 (0%)       0 (0%)           0 (0%)
  kube-system                auth-apikeys-qbg4k                                    200m (2%)     1 (12%)      300Mi (1%)       1Gi (6%)
  kube-system                auth-idp-rl8dj                                        300m (3%)     3200m (40%)  768Mi (4%)       3584Mi (22%)
  kube-system                auth-pap-s76p7                                        150m (1%)     1200m (15%)  456Mi (2%)       1536Mi (9%)
  kube-system                auth-pdp-jlb6f                                        600m (7%)     200m (2%)    768Mi (4%)       512Mi (3%)
  kube-system                calico-kube-controllers-d775694f-pzph9                250m (3%)     0 (0%)       100Mi (0%)       0 (0%)
  kube-system                calico-node-5xwb6                                     300m (3%)     0 (0%)       150Mi (0%)       0 (0%)
  kube-system                catalog-ui-vjj45                                      300m (3%)     300m (3%)    300Mi (1%)       300Mi (1%)
  kube-system                heapster-569fdfd65-ndvxh                              20m (0%)      0 (0%)       64Mi (0%)        0 (0%)
  kube-system                helm-api-6c9756484f-ql4vl                             350m (4%)     550m (6%)    556Mi (3%)       656Mi (4%)
  kube-system                helm-repo-5c8fcc8899-kd87g                            150m (1%)     200m (2%)    640Mi (4%)       640Mi (4%)
  kube-system                ibmcloud-image-enforcement-c558c6c95-xxfbx            128m (1%)     256m (3%)    128Mi (0%)       256Mi (1%)
  kube-system                icp-management-ingress-bdgt7                          200m (2%)     0 (0%)       256Mi (1%)       0 (0%)
  kube-system                icp-mongodb-0                                         0 (0%)        0 (0%)       0 (0%)           0 (0%)
  kube-system                image-manager-0                                       110m (1%)     0 (0%)       192Mi (1%)       0 (0%)
  kube-system                k8s-etcd-9.20.194.53                                  0 (0%)        0 (0%)       0 (0%)           0 (0%)
  kube-system                k8s-master-9.20.194.53                                5m (0%)       0 (0%)       10Mi (0%)        0 (0%)
  kube-system                k8s-proxy-9.20.194.53                                 0 (0%)        0 (0%)       0 (0%)           0 (0%)
  kube-system                kube-dns-n5qfk                                        100m (1%)     0 (0%)       70Mi (0%)        0 (0%)
  kube-system                logging-elk-filebeat-ds-z6xpm                         0 (0%)        0 (0%)       0 (0%)           0 (0%)
  kube-system                mariadb-0                                             500m (6%)     1 (12%)      128Mi (0%)       512Mi (3%)
  kube-system                metering-reader-vslkf                                 250m (3%)     0 (0%)       512Mi (3%)       0 (0%)
  kube-system                mgmt-repo-5cb9f9dc7b-thc28                            150m (1%)     200m (2%)    640Mi (4%)       640Mi (4%)
  kube-system                monitoring-prometheus-nodeexporter-xh787              0 (0%)        0 (0%)       0 (0%)           0 (0%)
  kube-system                nvidia-device-plugin-vm9m8                            150m (1%)     0 (0%)       0 (0%)           0 (0%)
  kube-system                platform-api-86dff555db-llbz2                         100m (1%)     100m (1%)    128Mi (0%)       512Mi (3%)
  kube-system                platform-deploy-749fc56fb7-cmjql                      100m (1%)     100m (1%)    128Mi (0%)       512Mi (3%)
  kube-system                platform-ui-qtvwm                                     300m (3%)     300m (3%)    256Mi (1%)       256Mi (1%)
  kube-system                secret-watcher-7994f75f9b-l4ffh                       0 (0%)        0 (0%)       0 (0%)           0 (0%)
  kube-system                service-catalog-apiserver-79nmn                       100m (1%)     100m (1%)    20Mi (0%)        200Mi (1%)
  kube-system                service-catalog-controller-manager-9c7bcf586-6kp2c    100m (1%)     100m (1%)    20Mi (0%)        200Mi (1%)
  kube-system                tiller-deploy-5677cc5dfb-m5k9h                        100m (1%)     0 (0%)       128Mi (0%)       0 (0%)
  kube-system                unified-router-zf4fs                                  20m (0%)      0 (0%)       64Mi (0%)        0 (0%)
  kube-system                web-terminal-55c549d48d-bn98q                         10m (0%)      100m (1%)    64Mi (0%)        512Mi (3%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests      Limits
  --------  --------      ------
  cpu       5043m (63%)   8906m (111%)
  memory    6846Mi (42%)  11852Mi (74%)
Events:
  Type    Reason     Age   From                  Message
  ----    ------     ----  ----                  -------
  Normal  NodeReady  55m   kubelet, 9.20.194.53  Node 9.20.194.53 status is now: NodeReady

Wednesday, 12 December 2018

DB2 on Windows - What a difference a CR/LF makes ....

Context - configuring DB2 on Windows to support an IBM Security Key Lifecycle Manager (SKLM) installation ….

So I start my instance: -

db2start

...
12/12/2018 10:48:22     0   0   SQL5043N  Support for one or more communications
 protocols specified in the DB2COMM environment variable failed to start success
fully. However, core database manager functionality started successfully.
SQL1063N  DB2START processing was successful.


Check the DB2 environment: -

db2set

...
DB2COMM=TCPIP
...

Check the DB2 Service Name: -

db2 get dbm config | find "SVCENAME"

...
 TCP/IP Service name                          (SVCENAME) = db2c_DB2
 SSL service name                         (SSL_SVCENAME) =
...

Checked the Windows TCP/IP services file: -

type c:\Windows\System32\drivers\etc\services | find "db2c_DB2"

...
db2c_DB2        50000/tcp
...

I then thought to check the DB2 diagnostics: -

db2diag -f

...
2018-12-12-10.48.20.943000+000 I296931F497          LEVEL: Error
PID     : 2524                 TID : 2204           PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
HOSTNAME: WIN-SDROBD85C4C
EDUID   : 2204                 EDUNAME: db2sysc 0
FUNCTION: DB2 UDB, common communication, sqlcctcpconnmgr, probe:5
MESSAGE : DIA3201E The service name "db2c_DB2" specified in the database
          manager configuration file cannot be found in the TCP/IP services
          file.



So I rechecked the TCP/IP services file, using Notepad: -

notepad.exe c:\Windows\System32\drivers\etc\services

and swiftly realised that, although I DID have the service listed: -

db2c_DB2        50000/tcp

there was NO Carriage Return / Line Feed ( CR/LF ) after the end of the line.

In other words, the terminator was after the p of tcp.

I added a CR/LF character and saved the file.

I then stopped the instance: -

db2stop

...
12/12/2018 10:50:59     0   0   SQL1064N  DB2STOP processing was successful.
SQL1064N  DB2STOP processing was successful.



and started it again: -

db2start

...
12/12/2018 10:51:25     0   0   SQL1063N  DB2START processing was successful.
SQL1063N  DB2START processing was successful.



which looked good.

I then checked the listening ports: -

netstat -aon|find "LISTEN"

...
TCP    0.0.0.0:49158          0.0.0.0:0              LISTENING       500
TCP    0.0.0.0:49159          0.0.0.0:0              LISTENING       1244
TCP    0.0.0.0:49163          0.0.0.0:0              LISTENING       492
TCP    0.0.0.0:49168          0.0.0.0:0              LISTENING       1360
TCP    0.0.0.0:49183          0.0.0.0:0              LISTENING       1308
TCP    0.0.0.0:50000          0.0.0.0:0              LISTENING       3664
TCP    127.0.0.1:53           0.0.0.0:0              LISTENING       1360
TCP    127.0.0.1:9633         0.0.0.0:0              LISTENING       1188

...

Oh frabjous day :-)

Java Versions - Compiled and Run-time

In the context of a client's Java application migration from Java 6 ( 1.6 ) to Java 8 ( 1.8 ), I wanted to show a colleague how we can typically run Java classes unmodified on Java 8, even though they were compiled using a much lower Java SDK ( JDK ).

I used two simple examples: -

Ferret


WebApp25


using this as inspiration: -


In both cases, I expanded the EAR / WAR files, and inspected one/more of the classes therein: -

mkdir /tmp/Ferret
cd  /tmp/Ferret
tar xvf ~/wlp/usr/servers/defaultServer/dropins/ferret-1.2.war

x META-INF/
x META-INF/MANIFEST.MF
x static/
x WEB-INF/
x WEB-INF/classes/
x WEB-INF/classes/net/
x WEB-INF/classes/net/wasdev/
x WEB-INF/classes/net/wasdev/samples/
x WEB-INF/classes/net/wasdev/samples/ferret/
x WEB-INF/classes/net/wasdev/samples/ferret/html/
x static/style.css
x WEB-INF/web.xml
x WEB-INF/classes/LICENSE
x WEB-INF/classes/net/wasdev/samples/ferret/FerretServlet.class
x WEB-INF/classes/net/wasdev/samples/ferret/html/Element.class
x WEB-INF/classes/net/wasdev/samples/ferret/html/Table.class
x WEB-INF/classes/net/wasdev/samples/ferret/html/Page.class
x WEB-INF/classes/net/wasdev/samples/ferret/html/Element$ElementType.class
x WEB-INF/classes/net/wasdev/samples/ferret/RequestData.class
x WEB-INF/classes/net/wasdev/samples/ferret/FerretData.class
x WEB-INF/ibm-web-ext.xml
x META-INF/maven/
x META-INF/maven/net.wasdev.wlp.sample/
x META-INF/maven/net.wasdev.wlp.sample/ferret/
x META-INF/maven/net.wasdev.wlp.sample/ferret/pom.xml
x META-INF/maven/net.wasdev.wlp.sample/ferret/pom.properties

...

javap -verbose WEB-INF/classes/net/wasdev/samples/ferret/FerretServlet.class|grep "major"

  major version: 51

mkdir /tmp/WebApp25
cd  /tmp/WebApp25
tar xvf ~/wlp/usr/servers/defaultServer/dropins/WebApp25EAR.ear

x META-INF/MANIFEST.MF
x META-INF/application.xml
x WebApp25.war


tar xvf WebApp25.war 

...
x META-INF/MANIFEST.MF
x WEB-INF/classes/com/ibm/fvt/filters/FilterWCSR001.class
x WEB-INF/classes/com/ibm/fvt/filters/FilterWCSR003.class
x WEB-INF/classes/com/ibm/fvt/filters/FilterWCSR007.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/CatchAll.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/ErrorPage.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/GetSession.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR001Error.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR001Forward.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR001Include.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR001Request.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR002.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR003.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR004.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR005.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR006.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR007.class
x WEB-INF/classes/com/ibm/ws/fvt/servlets/WCSR007a.class
x WEB-INF/ibm-web-bnd.xmi
x WEB-INF/ibm-web-ext.xmi
x WEB-INF/web.xml
x theme/Master.css
x WEB-INF/source/com/ibm/fvt/filters/FilterWCSR001.java
x WEB-INF/source/com/ibm/fvt/filters/FilterWCSR003.java
x WEB-INF/source/com/ibm/fvt/filters/FilterWCSR007.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/CatchAll.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/ErrorPage.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/GetSession.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR001Error.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR001Forward.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR001Include.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR001Request.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR002.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR003.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR004.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR005.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR006.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR007.java
x WEB-INF/source/com/ibm/ws/fvt/servlets/WCSR007a.java


javap -verbose WEB-INF/classes/com/ibm/ws/fvt/servlets/GetSession.class | grep "major"

  major version: 46

Using the lookup table from the Stackoverflow article: -

You're looking for this on the command line (for a class called MyClass):

On Unix/Linux:

javap -verbose MyClass | grep "major"
On Windows:

javap -verbose MyClass | findstr "major"
You want the major version from the results. Here are some example values:

• Java 1.2 uses major version 46
• Java 1.3 uses major version 47
• Java 1.4 uses major version 48
• Java 5 uses major version 49
• Java 6 uses major version 50
• Java 7 uses major version 51
• Java 8 uses major version 52
• Java 9 uses major version 53
• Java 10 uses major version 54
• Java 11 uses major version 55

I can see that Ferret ( version 51 ) was compiled for Java 7 and that WebApp25 ( version 46 ) was compiled for Java 1.2

And yet both run happily via the latest WebSphere Liberty Profile on Java 8…….

Nice.

Thursday, 6 December 2018

Doh, WebSphere Application Server, Resource Environment Providers and Not-so-Shared Libraries

I'm fiddling with old-school Java again, reminding myself how Resource Environment Providers work in WebSphere Application Server (WAS) 9.

Using this : -

Using resource environment providers in WebSphere Application Server

and this: -

Resource Environment Provider in EJB-jar xml

as inspiration, I'd written a very basic servlet, that reads from a Resource Environment Entry, setup using a Resource Environment Provider, ably provided by a Java class.

When I attempted to use my servlet ( the Resource Environment Entry is retrieved during the doGet() method, I saw this in the logs: -

...
[06/12/18 18:05:46:969 GMT] 00000098 annotation    W com.ibm.ws.webcontainer.annotation.WASAnnotationHelper inject(mo) SRVE8042E: An internal error caused the reference context that enables injection to not be initialized properly.
...
[06/12/18 18:05:47:539 GMT] 00000098 ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service Uncaught service() exception thrown by servlet com.davehay.Hello: java.lang.NoClassDefFoundError: com.ibm.acme.ree.lib.Config
at com.davehay.Hello.doGet(Hello.java:23)
...
Caused by: java.lang.ClassNotFoundException: com.ibm.acme.ree.lib.Config
...

Guess what ?

I bet you can't....

It was (l)user error.

Although I'd included my "utility" JAR as a Shared Library in WAS, I'd forgotten to "bind" it to the application : -



So, even though my application "knew" about the Resource Environment: -

it couldn't marry the two together i.e. the exception: -

...
Caused by: java.lang.ClassNotFoundException: com.ibm.acme.ree.lib.Config
...

took me right back to my beginnings with J2EE and WAS 2.X in the early noughts - it's always the class path, dummy

Tuesday, 4 December 2018

Using the Stream Editor (sed) with Bash variables

I've got a shell script to setup IBM HTTP Server (IHS), which includes a question: -

if [ -z "$1" ]
  then
    echo "For what product are you creating this ?"
    exit 1
fi
 and then uses the Stream Editor (sed) to use this variable : -

sed -i'' 's/PidFile\ logs/PidFile\ ${Product}\/logs/g' /opt/ibm/HTTPServer/${Product}/conf/httpd.conf
sed -i'' 's/ErrorLog\ logs/ErrorLog\ ${Product}\/logs/g' /opt/ibm/HTTPServer/${Product}/conf/httpd.conf
sed -i'' 's/CustomLog\ logs/CustomLog\ ${Product}\/logs/g' /opt/ibm/HTTPServer/${Product}/conf/httpd.conf
Sadly, this didn't quite work ....

Instead, I ended up with this: -
PidFile ${Product}/logs/httpd.pid
rather than this: -
PidFile WAS/logs/httpd.pid
 Thankfully, it was a quick fix - I just needed to change my Bash script to this: -

sed -i'' "s/PidFile\ logs/PidFile\ ${Product}\/logs/g" /opt/ibm/HTTPServer/${Product}/conf/httpd.conf
sed -i'' "s/ErrorLog\ logs/ErrorLog\ ${Product}\/logs/g" /opt/ibm/HTTPServer/${Product}/conf/httpd.conf
sed -i'' "s/CustomLog\ logs/CustomLog\ ${Product}\/logs/g" /opt/ibm/HTTPServer/${Product}/conf/httpd.conf

In other words, I replaced the single quotes around the s/from/to/g section of the Sed script with double quotes ....

Thanks to this: -

How do I use variables in a sed command?

for inspiration.

WebSphere Application Server (WAS) and Single Sign-On (SSO) - Back to 2011

It's been a while since last I did this, but I'm revisiting the happy days of 2011, when first I setup WAS and SSO with Microsoft Active Directory, Kerberos and SPNEGO.

I even presented upon the topic in 2012: -

IBM Connections and Desktop Single Sign-On using Microsoft Active Directory, Kerberos and SPNEGO

Whilst I've done the SPNEGO piece time and again, the thing that I'd not done in a while was to configure the fallback login page.

Thankfully, the IBM Connections documentation to which I referred back in 2012 is still there, albeit in more recent form: -

Configuring SPNEGO (and Kerberos optionally) on WebSphere Application Server

The significant thing is to create a fallback login page, using the HTML example here: -


This was then hosted on the IBM HTTP Server (IHS) service that fronts the WAS 9 environment: -

/opt/ibm/HTTPServer/htdocs/NoSpnegoRedirect.html

which I then configured in WAS: -


Now, when I hit my servlet via a clean browser: -

https://was.uk.ibm.com:8443/HelloAgain/HelloAgain

I automatically get a login page: -

Note that the URL automatically gets ?noSPNEGO : -

https://was.uk.ibm.com:8443/HelloAgain/login.html?noSPNEGO

thanks to the filter criteria: -



WebSphere Application Server - Get your security roles right

I'm back in the early noughties, tinkering with Java Enterprise Edition (JEE) servlets and WebSphere Application Server (WAS) Network Deployment (ND).

This, in part, is to enable me to test my SPNEGO / Kerberos configuration.

Having created a JEE application in Eclipse, with servlet, login page etc. and having tested it in Liberty, I'd deployed my code to WAS ND.

However, when I tried to access the servlet: -

https://was.uk.ibm.com:9444/HelloWorld/Hello

I saw this: -

[03/12/18 17:15:19:295 GMT] 00000096 WebCollaborat A   SECJ0129E: Authorization failed for user hayd:defaultWIMFileBasedRealm while invoking GET on default_host://HelloWorld, Authorization failed, Not granted any of the required roles: reader

I'd already ensured that I'd mapped my users ( hayd and Administrator ) to the appropriate role: -



However, I was holding it wrong .... when I'd hand-cranked the web.xml, I'd made a little booboo.

I'd specified the Security Role as: -

Reader

but specified the Auth Constraint -> Role Name as: -

reader

So it made NO difference to which user/group I assigned the Reader role; it'd never work :-(

Get it right .....

SPNEGO - Not working, now working

There's a theme to my blog posts these days: -

Docker for macOS - Broken, now fixed 

Maven and Spring and Liberty - Broken, now fixed 

Kerberos and Red Hat Enterprise Linux - Now friends

so this one is all about SPNEGO and Kerberos.

Having enabled WebSphere Application Server (WAS) 9 and Windows Server 2012 R2 to play nicely together with Kerberos/SPNEGO, I was STILL seeing a logon box in Internet Explorer 11: -




I'd already checked that Integrated Windows Authentication (IWA) was enabled; this now appears to be the default: -




*BUT* guess what I'd forgotten ?

Yep, I had NOT added the site: -

https://was.uk.ibm.com

to the appropriate zoneS in IE.

Now I'd already realised that IE11 no longer shows the internet/intranet/trusted zone in the status bar, BUT you can get a view using the [Alt] [F] [R] key sequence: -




Initially I added the site to the Trusted sites zone: -



but to no avail.

I then followed this: -

Configuring the client browser to use SPNEGO

and added the site to the Local intranet zone: -



which actually moves it from Trusted sites: -



and now it just works.


Nice :-)