IBM Operational Decision Manager Rules - Problems augmenting profiles

So I've been battling with an issue today, whilst also wrestling with SSL/TLS ciphers and wrangling and munging Java class loader issues.

Essentially, I've installed IBM Operational Decision Manager (ODM) Advanced Decision Server onto my Linux VM.

I'm then creating the necessary WebSphere Application Server (WAS) profiles to build out an ODM cell.

Here's the profile creation part: -

/opt/ibm/WebSphere/AppServer/bin/manageprofiles.sh -create -templatePath /opt/ibm/WebSphere/AppServer/profileTemplates/management -profileName Dmgr01 -nodeName Dmgr -isDefault -enableAdminSecurity true -adminUserName wasadmin -adminPassword passw0rd

/opt/ibm/WebSphere/AppServer/bin/manageprofiles.sh -create -templatePath /opt/ibm/WebSphere/AppServer/profileTemplates/managed -profileName AppSrv01 -nodeName Node1

I then start the Deployment Manager: -

/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/bin/startManager.sh

and add the node into the cell: -

/opt/ibm/WebSphere/AppServer/profiles/AppSrv01/bin/addNode.sh `hostname` 8879 -profileName AppSrv01 -username wasadmin -password passw0rd

Finally ( for now ) I stop the Deployment Manager: -

/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/bin/stopManager.sh -username wasadmin -password passw0rd

before augmenting the DM profile: -

/opt/ibm/WebSphere/AppServer/bin/manageprofiles.sh -augment -profileName Dmgr01 -templatePath /opt/ibm/WebSphere/AppServer/profileTemplates/rules/management/ds/

This bit failed with: -

<record>
    <date>2015-10-30T04:28:58</date>
    <millis>1446222538186</millis>
    <sequence>41</sequence>
    <logger>com.ibm.wsspi.profile.WSProfileTemplate</logger>
    <level>INFO</level>
    <class>com.ibm.wsspi.profile.WSProfileTemplate</class>
    <method>parseTemplateMetadata</method>
    <thread>0</thread>
    <message>Unexpected exception caught!</message>
</record>
<record>
    <date>2015-10-30T04:28:58</date>
    <millis>1446222538186</millis>
    <sequence>42</sequence>
    <logger>com.ibm.wsspi.profile.WSProfileTemplate</logger>
    <level>SEVERE</level>
    <class>com.ibm.wsspi.profile.WSProfileTemplate</class>
    <method>parseTemplateMetadata</method>
    <thread>0</thread>
    <message>Exception caught while parsing template metadata: null</message>
</record>
<record>
    <date>2015-10-30T04:28:58</date>
    <millis>1446222538186</millis>
    <sequence>43</sequence>
    <logger>com.ibm.ws.profile.utils.PrereqTemplateUtils</logger>
    <level>INFO</level>
    <class>com.ibm.ws.profile.utils.PrereqTemplateUtils</class>
    <method>getListOfTemplatePathsForStackAugment</method>
    <thread>0</thread>
    <message>[/opt/ibm/WebSphere/AppServer/profileTemplates/rules/management/ds/] is not a valid profile template for Dmgr01.</message>
</record>

This is what I had installed: -

/opt/ibm/InstallationManager/eclipse/tools/imcl listInstalledPackages -features

com.ibm.cic.agent_1.8.2000.20150303_1526 : 
com.ibm.websphere.ND.v85_8.5.5007.20150820_2101 : com.ibm.sdk.6_64bit,ejbdeploy,embeddablecontainer,samples,thinclient
com.ibm.websphere.IHS.v85_8.5.5007.20150820_2101 : arch.64bit
com.ibm.websphere.PLG.v85_8.5.5007.20150820_2101 : com.ibm.jre.6_64bit
com.ibm.websphere.odm.ds.rules.v87_8.7.0.20141114_0949 : com.ibm.wds.rules.res.feature

/opt/ibm/InstallationManager/eclipse/tools/imcl listInstalledPackages -features -long

/opt/ibm/InstallationManager/eclipse : com.ibm.cic.agent_1.8.2000.20150303_1526 : IBM® Installation Manager : 1.8.2 : 
/opt/ibm/WebSphere/AppServer : com.ibm.websphere.ND.v85_8.5.5007.20150820_2101 : IBM WebSphere Application Server Network Deployment : 8.5.5.7 : com.ibm.sdk.6_64bit,ejbdeploy,embeddablecontainer,samples,thinclient
/opt/ibm/HTTPServer : com.ibm.websphere.IHS.v85_8.5.5007.20150820_2101 : IBM HTTP Server for WebSphere Application Server : 8.5.5.7 : arch.64bit
/opt/ibm/WebSphere/Plugins : com.ibm.websphere.PLG.v85_8.5.5007.20150820_2101 : Web Server Plug-ins for IBM WebSphere Application Server : 8.5.5.7 : com.ibm.jre.6_64bit
/opt/ibm/ODM87 : com.ibm.websphere.odm.ds.rules.v87_8.7.0.20141114_0949 : Decision Server Rules : 8.7.0.0 : com.ibm.wds.rules.res.feature

using this response file: -

...

  <profile id='Operational Decision Manager V8.7' installLocation='/opt/ibm/ODM87'>
    <data key='eclipseLocation' value='/opt/ibm/ODM87'/>
    <data key='user.import.profile' value='false'/>
    <data key='cic.selector.os' value='linux'/>
    <data key='cic.selector.arch' value='x86_64'/>
    <data key='cic.selector.ws' value='gtk'/>
    <data key='cic.selector.nl' value='en'/>
    <data key='user.lic.dsr' value='full'/>
    <data key='user.prod.dsr' value='false'/>
    <data key='user.wodm_was_home' value='/opt/ibm/WebSphere/AppServer'/>
    <data key='user.lic.dsr' value='full'/>
    <data key='user.prod.dsr' value=''/>
    <data key='user.lic.dc' value='full'/>
    <data key='user.wodm_admin_username' value='wasadmin'/>
    <data key='user.wodm_admin_password' value='passw0rd'/>
    <data key='user.confirm_password' value='passw0rd'/>
  </profile>
  <install modify='false'>
    <offering profile='Operational Decision Manager V8.7' id='com.ibm.websphere.odm.ds.rules.v87' version='8.7.0.20141114_0949' features='com.ibm.wds.jdk.feature,base,com.ibm.wds.rules.res.feature' installFixes='none'/>
  </install>
...

To the best of my knowledge / experience, that should've been all I needed ( from a pure ODM perspective ).

And yet .....

I dug back into a prior blog post: -

Adding Ant to IBM Operational Decision Manager

which is what had led me to install this particular component of ODM: -

I guessed that I might be missing the appropriate WAS template for ODM, so I then installed the Decision Server Rules profile templates for WebSphere Application Server  8.7.0.0 element: -

/opt/ibm/InstallationManager/eclipse/tools/imcl listInstalledPackages -features

com.ibm.cic.agent_1.8.2000.20150303_1526 : 
com.ibm.websphere.ND.v85_8.5.5007.20150820_2101 : com.ibm.sdk.6_64bit,ejbdeploy,embeddablecontainer,samples,thinclient
com.ibm.websphere.IHS.v85_8.5.5007.20150820_2101 : arch.64bit
com.ibm.websphere.PLG.v85_8.5.5007.20150820_2101 : com.ibm.jre.6_64bit
com.ibm.websphere.odm.ds.rules.v87_8.7.0.20141114_0949 : com.ibm.wds.rules.res.feature
com.ibm.websphere.odm.pt.rules.v87_8.7.0.20141114_1039 : 

/opt/ibm/InstallationManager/eclipse/tools/imcl listInstalledPackages -features -long

/opt/ibm/InstallationManager/eclipse : com.ibm.cic.agent_1.8.2000.20150303_1526 : IBM® Installation Manager : 1.8.2 : 
/opt/ibm/WebSphere/AppServer : com.ibm.websphere.ND.v85_8.5.5007.20150820_2101 : IBM WebSphere Application Server Network Deployment : 8.5.5.7 : com.ibm.sdk.6_64bit,ejbdeploy,embeddablecontainer,samples,thinclient
/opt/ibm/HTTPServer : com.ibm.websphere.IHS.v85_8.5.5007.20150820_2101 : IBM HTTP Server for WebSphere Application Server : 8.5.5.7 : arch.64bit
/opt/ibm/WebSphere/Plugins : com.ibm.websphere.PLG.v85_8.5.5007.20150820_2101 : Web Server Plug-ins for IBM WebSphere Application Server : 8.5.5.7 : com.ibm.jre.6_64bit
/opt/ibm/ODM87 : com.ibm.websphere.odm.ds.rules.v87_8.7.0.20141114_0949 : Decision Server Rules : 8.7.0.0 : com.ibm.wds.rules.res.feature
/opt/ibm/ODM87 : com.ibm.websphere.odm.pt.rules.v87_8.7.0.20141114_1039 : Decision Server Rules profile templates for WebSphere Application Server  : 8.7.0.0 : 

via another IIM response file: -

...

  <profile id='Operational Decision Manager V8.7' installLocation='/opt/ibm/ODM87'>
    <data key='eclipseLocation' value='/opt/ibm/ODM87'/>
    <data key='user.import.profile' value='false'/>
    <data key='cic.selector.os' value='linux'/>
    <data key='cic.selector.arch' value='x86_64'/>
    <data key='cic.selector.ws' value='gtk'/>
    <data key='cic.selector.nl' value='en'/>
    <data key='user.lic.dsr' value='full'/>
    <data key='user.prod.dsr' value='false'/>
    <data key='user.wodm_was_home' value='/opt/ibm/WebSphere/AppServer'/>
    <data key='user.lic.dc' value='full'/>
    <data key='user.wodm_admin_username' value='wasadmin'/>
    <data key='user.wodm_admin_password' value='passw0rd'/>
    <data key='user.confirm_password' value='passw0rd'/>
  </profile>
  <install modify='false'>
    <!-- Decision Server Rules profile templates for WebSphere Application Server  8.7.0.0 -->
    <offering profile='Operational Decision Manager V8.7' id='com.ibm.websphere.odm.pt.rules.v87' version='8.7.0.20141114_1039' features='main.feature' installFixes='none'/>
  </install>
...

Now we're good to go ...

/opt/ibm/WebSphere/AppServer/bin/manageprofiles.sh -augment -profileName Dmgr01 -templatePath /opt/ibm/WebSphere/AppServer/profileTemplates/rules/management/ds/

INSTCONFSUCCESS: Profile augmentation succeeded.

So that's all good then .....

I do need to work out the differences but that's a job for another day ...

IBM Business Process Manager - What Goes In What Databases ?

My client asked me this, so I thought I'd share it here.

Context

In the world of IBM BPM Advanced 8.5, with DB2, we have three (five) distinct databases.

Most typically, we have the Process Server database ( often known as the BPM Database, BPMDB ), the Shared Database ( often known as the Common Database, CMNDB ) and the Performance Database ( often known as the Performance Data Warehouse Database, PDWDB ).

Optionally, we can have two more databases; the Messaging Engine Database ( MEDB ) and the Business Process Choreographer Database ( BPCDB ).

The IBM Redbook, Business Process Management Deployment Guide Using IBM Business Process Manager V8.5, puts this so much better: -

Most of the time, I use the three DB topology pattern, which is more-or-less the default. This separation allows me to put different databases in different places, perhaps on different disks ( LUNs ), user IDs ( instances ), different servers etc.

For the record, I'm a DB2 guy by trade, so I realise that, in the world of Oracle, we have a single database ( Oracle itself ) and multiple schemas.

But the principle remains much the same ...

New iCloud Terms and Conditions

I kept seeing this on an iPhone 5S after an update to iOS 9.1: -

New iCloud Terms and Conditions

To use iCloud on this iPhone you must accept the new Terms and Conditions.

However, each time I clicked on View Terms I merely got taken to the iCloud page in Settings, which made no difference.

I tried Turning It Off And On Again but to no avail.

I then Google'd about for a while, and found a post suggesting that I needed to go into Settings > Mail, Contacts, Calendars.

I then selected the iCloud mail account and ... saw a Terms and Conditions link.

I selected this, read the Ts & Cs, accepted them and .... we're now good to go :-)

Which is nice :-)

IBM Integration Bus v10 self study labs

This via Twitter: -

IBM Integration Bus v10 self study labs

IBM® Integration Bus V10 betaworks labs have been published and can be found here. Each lab comprises one or more guides as PDF files, and is usually accompanied by an archive file (.zip file) that you can use to complete the lab activities. These self study labs provide the opportunity to develop your skills in IBM Integration Bus.

Please let us know if there are any areas you would like to see covered by a self study lab.

IBM Integration Bus v10 self study labs

IBM Operational Decision Manager 8.7.1 Fixpack 1

For the record, IBM shipped the latest fix pack for ODM 8.7.1 last week, known ( by me, at least ) as 8.7.1.1.

The fix pack is available on IBM Fix Central.

Here's a short-cut to the fix pack itself: -

http://www.ibm.com/support/fixcentral/swg/quickorder?parent=ibm%7EWebSphere&product=ibm/WebSphere/WebSphere+Operational+Decision+Management&release=8.7.1.0&platform=All&function=all&source=fc

Learning Jython - Every Day is a School Day

One of my fiends was looking for some Jython/Python guidance, so I sent him this -

Learning Python with Raspberry Pi 

WebSphere Application Server Administration Using Jython

WebSphere Application Server Administration using Jython

Using Jython Scripting Language With WSADMIN

Sample Scripts for WebSphere Application Server

Scripting from scratch: Creating a Jython administrative script for IBM WebSphere Application Server

Using the latest Jython with a WebSphere Application Server wsadmin thin client

IBM WOMBAT - Project Artifact Analyzer for IBM BPM and IBM Business Monitor

I've used this tool before it was available outside of IBM.

It's of immense value when analysing BPM projects, I was specifically looking at TWX exports from Process Center.

And it's on Bluemix ... what's not to like ?

IBM WOMBAT - Project Artifact Analyzer for IBM BPM and IBM Business Monitor

IBM UrbanCode Deploy - Increase security when impersonating other users

I found this via IBM developerWorks on Twitter: -

Increase security when impersonating other users

Save time with the User Impersonation feature in IBM UrbanCode Deploy

Use IBM UrbanCode Deploy's User Impersonation feature to facilitate deployment of more complex applications where specific features are managed under dedicated users and specific deployment steps need to be executed as these users.

Hmmm, Process Portal Not Playing Nicely - IBM BPM Advanced 8.5.5.0

Problem

Hitting Process Portal, either via IHS ( on port 8443 ) or directly ( on port 9443 / 9446 ) results in a garbled page, with most of the artefacts down the left-hand side of the page.
Problem only seen when one explicitly logs into Process Portal

Browser ( IE or Firefox ) makes no difference.

Direct - via AppClusterMember1

https://bpm855.uk.ibm.com:9443/ProcessPortal/jsp/index.jsp

Direct - via AppClusterMember2

https://bpm855.uk.ibm.com:9446/ProcessPortal/jsp/index.jsp

IHS

https://bpm855.uk.ibm.com:8443/ProcessPortal/jsp/index.jsp

Exceptions seen in AppClusterMemberX SystemOut.log

...
[19/10/15 10:13:37:834 BST] 000001c9 JMSClassifica W   XJMS0021E: Unable to map destination and bus to workclass. Aborting the xd classification filter for the current request. Request context information is com.ibm.ws.xd.jms.proxy.impl.AsynchJMSServiceContextImpl@b7558001<bus=BPM.PSCell1De1.Bus,destination=PortalWebMessagingTopicSpace.AppCluster>.
...
[19/10/15 10:18:27:625 BST]     FFDC Exception:java.io.IOException SourceId:com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream
ProbeId:102 Reporter:com.ibm.ws.webcontainer.channel.WCCByteBufferInputStream@af8277d8
java.io.IOException: Async IO operation failed (1), reason: RC: 76  Socket is not connected
        at com.ibm.io.async.AsyncLibrary$IOExceptionCache.<init>(AsyncLibrary.java:924)
        at com.ibm.io.async.AsyncLibrary$IOExceptionCache.get(AsyncLibrary.java:937)
...
[19/10/15 11:43:34:983 BST] 0000014a JSONTransport W org.cometd.server.transport.LongPollingTransport handleJSONParseException Error
parsing JSON:
                                 java.lang.RuntimeException: java.io.IOException: Async IO operation failed (3), reason: RC: 76  Socket is not connected
        at org.eclipse.jetty.util.ajax.JSON$ReaderSource.getNext(JSON.java:1519)
        at org.eclipse.jetty.util.ajax.JSON$ReaderSource.hasNext(JSON.java:1486)
        at org.eclipse.jetty.util.ajax.JSON.parse(JSON.java:789)
        at org.cometd.common.JettyJSONContext.parse(JettyJSONContext.java:57)
        at org.cometd.server.AbstractServerTransport.parseMessages(AbstractServerTransport.java:154)
        at org.cometd.server.transport.JSONTransport.parseMessages(JSONTransport.java:72)
        at org.cometd.server.transport.LongPollingTransport.handle(LongPollingTransport.java:192)
        at org.cometd.server.CometdServlet.service(CometdServlet.java:166)
        at com.ibm.bpm.socialbus.servlet.SocialBusServiceServlet.service(SocialBusServiceServlet.java:183)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1230)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:779)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
        at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:178)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.invokeTarget(WebAppFilterChain.java:136)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:97)
        at com.ibm.bpm.servlet.filters.GenericSecurityServletFilter.doFilter(GenericSecurityServletFilter.java:84)
        at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:195)
        at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:964)
        at com.ibm.ws.webcontainer.filter.WebAppFilterManager.invokeFilters(WebAppFilterManager.java:1104)
        at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3901)
        at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:304)
        at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:981)
        at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1662)
        at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:200)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:463)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:530)
...

Potential Indication of Root Cause - java.io.IOException: Async IO operation failed (1), reason: RC: 76  Socket is not connected

Feels like client-server issue, perhaps problem at IP stack

Definitely red herring - known "problem" covered in multiple Technotes

[19/10/15 11:17:04:847 BST] 00000141 JMSClassifica W   XJMS0021E: Unable to map destination and bus to workclass. Aborting the xd classification filter for the current
request. Request context information is

com.ibm.ws.xd.jms.proxy.impl.AsynchJMSServiceContextImpl@fe11fe03<bus=BPM.PSCell1De1.Bus,destination=PortalWebMessagingTopicSpace.AppCluster>.

A XJMS0021E error occurs in the SystemOut.log file for IBM Business Process Manager (BPM) Version 8.5

XJMS0021E destination and bus workclass problem is logged to WAS systemOUT

Further Analysis

Used Firefox Debugger ( [Fn] [F12] )

Solution

Force BusinessSpace to reload BusinessSpace - templates, spaces and themes

Edit: -

/opt/ibm/WebSphereProfiles/PSCell1AppSrv01/BusinessSpace/AppCluster/mm.runtime.prof/public/oobLoadedStatus.properties

and change from: -

#Mon Oct 19 14:08:27 BST 2015
importTemplates.txt=false
importSpaces.txt=false
importThemes.txt=false

to: -

#Mon Oct 19 14:08:27 BST 2015
importTemplates.txt=true
importSpaces.txt=true
importThemes.txt=true

and restart AppCluster

( File is then reset back to false setting once cluster has been restarted - once templates, spaces etc. have been reset to their OOB defaults )

Problem does *NOT* appear to recur, including java.io.IOException exception

Re-tested Process Portal ( both direct and via IHS )

Direct - via AppClusterMember1

https://bpm855.uk.ibm.com:9443/ProcessPortal/jsp/index.jsp

Direct - via AppClusterMember2

https://bpm855.uk.ibm.com:9446/ProcessPortal/jsp/index.jsp

IHS

https://bpm855.uk.ibm.com:8443/ProcessPortal/jsp/index.jsp

Also tested Business Space

Direct - via AppClusterMember1

https://bpm855.uk.ibm.com:9443/mum/enabler

Direct - via AppClusterMember2

https://bpm855.uk.ibm.com:9446/mum/enabler

IHS

https://bpm855.uk.ibm.com:8443/mum/enabler

IBM HTTP Server - Working with Default Certificates issued by a Certificate Authority

I'm working through a process to have IBM HTTP Server (IHS) and WebSphere Application Server (WAS) use certificates minted by a Certificate Authority, specifically Microsoft Windows Server 2008, rather than using so-called self-signed certificates.

In the past, the bulk of my experience has been using self-signed certificates in IHS, although I did do some work with a client ~4-5 years ago where we were using CA-signed certificates.

Equally, I've only ever had WAS act as a certificate signer, which has been the default for ages. In this regard, WAS is the signer, and takes care of issuing and replacing certificates throughout the cell.

However, on my current project, we have a requirement to use CA-signed certificates throughout the estate.

Therefore, whilst I've made some great progress with WAS using Windows CA-signed certificates ( about which I'll blog later ), I did hit a roadblock with IHS.

I started by creating a Certificate Signer Request (CSR) as follows: -

/opt/ibm/HTTPServer/bin/gskcapicmd -certreq -create -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd -label wasdemo.uk.ibm.com -dn cn=wasdemo.uk.ibm.com -file /home/wasadmin/wasdemo.uk.ibm.com.arm -size 2048 -sigalg SHA256WithRSA

and sent the resulting wasdemo.uk.ibm.com.arm file to my Windows Server box.

Once I'd created the certificate, I got back a file from the CA in the PKCS7 format - certnew.p7b.

Initially I imported this file into IHS' key store, I was able to list it: -

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

Certificates found
* default, - personal, ! trusted, # secret key
- wasdemo.uk.ibm.com

which was fine.

I then set the certificate to be the default: -

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -setdefault -label wasdemo.uk.ibm.com -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

However, when I attempted to validate that the certificate was default, I still saw this: -

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

Certificates found
* default, - personal, ! trusted, # secret key
- wasdemo.uk.ibm.com

and, worse still, this was what I saw when I queried the default certificate within the key store: -

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -getdefault -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

CTGSK3029W The database does not contain a certificate with label "GSKCAPICMD_TXT_DEFAULT" 

-Command usage-
-db                   Required
-pw | -stashed        Optional
-type                 Optional <cms | kdb | pkcs12 | p12>
 
Long story short, it was a PEBCAK.

I was doing it wrong ....

This was what I should've done ....

Take it from the top ....

Create the Keystore

/opt/ibm/HTTPServer/bin/gskcapicmd -keydb -create -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd -type cms -expire 3650 -stash

Create the CSR

/opt/ibm/HTTPServer/bin/gskcapicmd -certreq -create -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd -label wasdemo.uk.ibm.com -dn cn=wasdemo.uk.ibm.com -file /home/wasadmin/wasdemo.uk.ibm.com.arm -size 2048 -sigalg SHA256WithRSA

List the Certificates - of which there should be none

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

No certificates were found.

Add the root certificate ( obtained from the Windows box )

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -add -file certnew.cer -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

List the Certificates - should now show the root certificate

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

Certificates found
* default, - personal, ! trusted, # secret key
! CN=uk-WIN-AJ9S32NP29C-CA,DC=uk,DC=ibm,DC=com

Note the symbol ( ! ) to show that the root certificate is trusted

Receive the Personal Certificate for IHS

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -receive -file certnew.p7b -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

List the Certificates - should now show both certificates

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

Certificates found
* default, - personal, ! trusted, # secret key
! CN=uk-WIN-AJ9S32NP29C-CA,DC=uk,DC=ibm,DC=com
- wasdemo.uk.ibm.com

Note the symbol ( - ) to show that the certificate is personal

Also note that the the personal certificate is NOT shown as default - it has no asterisk ( * ) symbol next to it

Set the personal certificate to be default

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -setdefault -label wasdemo.uk.ibm.com -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

List the Certificates - should now show trusted root and default personal certificate

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

Certificates found
* default, - personal, ! trusted, # secret key
! CN=uk-WIN-AJ9S32NP29C-CA,DC=uk,DC=ibm,DC=com
*- wasdemo.uk.ibm.com

Validate the Default Certificate

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -getdefault -db /opt/ibm/HTTPServer/ssl/keystore.kdb -pw passw0rd

Label : wasdemo.uk.ibm.com
Key Size : 2048
Version : X509 V3
Serial : 125e65d100000000001d
Issuer : CN=uk-WIN-AJ9S32NP29C-CA,DC=uk,DC=ibm,DC=com
Subject : CN=wasdemo.uk.ibm.com
Not Before : 17 October 2015 19:32:02 GMT+01:00
Not After : 16 October 2017 19:32:02 GMT+01:00
Fingerprint : 
b6c9614bd8f7c6747040b9d5450d143d
a028d430

Now when I start IHS and access it via Internet Explorer from the Windows server itself: -

https://wasdemo.uk.ibm.com:8443/

it all works and, more importantly, the browser trusts IHS and doesn't show any exceptions re untrusted certificates.

Which is nice ...

IBM Business Process Manager operation overview, Part 3: Advanced operations

Part 3 of this series introduces advanced operations tasks that administrators for IBM® Business Process Manager (IBM BPM) complete in their daily operation work. The maintenance of a clustered server environment for IBM BPM that interacts with many back-end and front-end systems and services can be a challenging task. This final part in the series guides administrators through advanced operations such as troubleshooting, performance tuning, disaster recovery, and high availability - including an operation checklist.

http://www.ibm.com/developerworks/bpm/library/techarticles/1510_chan3-trs/1510_chan3.html

Fascinating - Internet Explorer and the Lack of Transport Layer Security

I've just spent a happy 10 minutes trying to work out why Internet Explorer 8 wouldn't load a secure web page from WebSphere Application Server, specifically the Integrated Solutions Console (ISC).

This is what I saw from IE: -

whereas Firefox was more than happy to render the page: -

Then I thought back to something a colleague had said earlier in the week ....

I'm using TLS 1.2 as my default encryption in WAS ( see an earlier blog post - WebSphere Application Server - Tinkering with Property Files and a spot of SSL/TLS ) and, therefore, WAS is expecting any client, including a browser, to support TLS 1.2.

Firefox confirms the encryption : -

Thinking back to what my colleague had said, I checked IE's settings: -

and, guess what ....

For the record, I am experimenting with the replacement of WAS' own CA-managed certificates ( yes, WAS can/does act as a Certificate Authority for certificates used inside the cell ) with those generated by Active Directory.

I'll write this up more fully in the future, but one of the net effects is that Internet Explorer, running in the same AD domain, now trusts the WAS HTTPS-secured pages, because they share the same CA signer.

Which is nice ....

*UPDATE*

For the record, WAS was trying to tell me what was going wrong, with: -

...
[15/10/15 15:22:39:130 BST] 000000d4 SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported

...

Caused by: javax.net.ssl.SSLHandshakeException: Client requested protocol TLSv1 not enabled or not supported

...

[15/10/15 15:24:51:003 BST] 000000d5 SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLHandshakeException: Client requested protocol SSLv3 not enabled or not supported

...

Caused by: javax.net.ssl.SSLHandshakeException: Client requested protocol SSLv3 not enabled or not supported

...

if only I'd looked in SystemOut.log :-)

WebSphere Application Server - Tinkering with Property Files and a spot of SSL/TLS

This is in the context of my ongoing voyage of discovery with Transport Layer Security (TLS) 1.2 within a WebSphere Application Server (WAS) 8.5.5.7 cell.

It is, for this reason, that I was reading this: -

WebSphere Application Server and Liberty Security Update

Specifically, having enabled TLS 1.2 as the default protocol for my cell: -

/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

cellID=AdminControl.getCell()

AdminTask.modifySSLConfig('[-alias CellDefaultSSLSettings -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+' -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers ]')

As soon as I save and synchronise the changes: -

AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

I see this exception: -

ADMS0206I: The configuration synchronization failed for node: AppSrv01Node.

in the Deployment Manager's SystemOut.log file, and this: -

[08/10/15 15:19:45:430 BST]     FFDC Exception:javax.net.ssl.SSLHandshakeException SourceId:com.ibm.ws.management.filetransfer.client.FileTransferClientImpl.download ProbeId:1575 Reporter:com.ibm.ws.management.filetransfer.client.FileTransferClientImpl@a36b36a1
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

...

Caused by: java.io.EOFException: SSL peer shut down incorrectly

in the Node Agent's FFDC log.

"Talking" this through with the wider WebSphere team, it was pointed out that, by default, I have Dynamic SSL Configuration enabled: -

which means that, as soon as I committed the change, the Deployment Manager immediately starting using AND enforcing TLS 1.2, whereas the Node Agent(s) continued using SSL/TLS.

I could see this by comparing/contrasting the cell and node-level security.xml files: -

diff AppSrv01/config/cells/WASCell/security.xml Dmgr01/config/cells/WASCell/security.xml

108c108
<     <setting xmi:id="SecureSocketLayer_1" clientAuthentication="false" securityLevel="HIGH" enabledCiphers="" jsseProvider="IBMJSSE2" sslProtocol="SSL_TLS" keyStore="KeyStore_1" trustStore="KeyStore_2" trustManager="TrustManager_2" keyManager="KeyManager_1"/>
---
>     <setting xmi:id="SecureSocketLayer_1" clientAuthentication="false" securityLevel="HIGH" enabledCiphers="" jsseProvider="IBMJSSE2" sslProtocol="TLSv1.2" keyStore="KeyStore_1" trustStore="KeyStore_2" trustManager="TrustManager_2" keyManager="KeyManager_1"/>

where the first is from the node and the second is from the DM.

Bill's presentation ( see the earlier link ) explicitly warns about this, on page 56: -

WARNING: BE SURE TO DISABLE dynamic SSL Prior to turning on strict mode. The Dmgr and Nodes will be in a incompatible mode. The Dmgr will likely switch to TLSv1.2 before the nodes do, they will likely still be at SSL_TLS, and the nodes and dmgr will no long be able to communicate. We STRONGLY RECOMMEND that you stop all the nodes except the Dmgr. Do the conversion on the console, restart the Dmgr, manually sync the nodes, then start the node agents and servers.

To start with, I experimented by disabling Dynamic SSL Configuration.

Whilst I knew that I could manually modify security.xml I was looking for a programmatic / scripted way of achieving this.

This is what I did: -

configProperties=AdminConfig.list('Cell')
AdminTask.extractConfigProperties(configProperties, '[-propertiesFileName /tmp/davehay.properties -filterMechanism SELECTED_SUBTYPES -selectedSubTypes [Security ] ]')

The resulting file - /tmp/davehay.properties - includes the line: -

dynamicallyUpdateSSLConfig=true #boolean,default(false)

Therefore, I created a pair of property files: -

dynamic_off.properties

# SubSection 1.0 # Security Section
#
ResourceType=Security
ImplementingResourceType=Security
ResourceId=Cell=!{cellName}:Security=ID#Security_1
#

#
#Properties
#
dynamicallyUpdateSSLConfig=false #boolean,default(false)

dynamic_on.properties

# SubSection 1.0 # Security Section
#
ResourceType=Security
ImplementingResourceType=Security
ResourceId=Cell=!{cellName}:Security=ID#Security_1
#

#
#Properties
#
dynamicallyUpdateSSLConfig=true #boolean,default(false)

which allowed me to turn off Dynamic Security Configuration: _

AdminTask.validateConfigProperties('-propertiesFileName /tmp/dynamic_off.properties')

AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

which returns 'true'

and then turn it back on again: -

AdminTask.validateConfigProperties('-propertiesFileName /tmp/dynamic_off.properties')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

which again returns 'true' :-)

However, I realised that I was going against the recommendation from Bill i.e. to ONLY have the Deployment Manager running, make the SSL_TLS > TLSv1.2 change and then manually synchronise the node(s).

This is what I did, in summary: -

------------------------------------------------------------------------------------

(a) Fully syncronise the cell

(b) Stop the Node Agent

(c) Modify the SSL configuration: -

AdminTask.modifySSLConfig('[-alias CellDefaultSSLSettings -scopeName (cell):WASCell -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):WASCell -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):WASCell -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers ]')

(d) Update the Node's ssl.client.props file to reflect the fact that we're now using TLS v1.2 rather than SSL_TLS

vi /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/properties/ssl.client.props

changing from: -

com.ibm.ssl.protocol=SSL_TLS

to: -

com.ibm.ssl.protocol=TLSv1.2

(e) Synchronise the Node with the cell: -

 /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/bin/syncNode.sh `hostname` -user wasadmin -password passw0rd

(f) Validate that DM and Node have consistent SSL/TLS settings: -

diff /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/config/cells/WASCell/security.xml /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/config/cells/WASCell/security.xml

(g) Start the node

(h) Validate that I can see the node from the DM

(i) Mix a Margherita     

------------------------------------------------------------------------------------

So, what did I learn ?

Firstly, read Bill's document from cover to cover.

Secondly, remember Dynamic SSL Configuration.

Thirdly, I now know how to work with WAS property files in Jython

Fourthly, I like Margheritas

WebSphere Application Server and Liberty Security Update

This is what I'm reading right now, in the context of enabling Transport Layer Security (TLS) 1.2 within a WebSphere Application Server Network Deployment 8.5.5.7 infrastructure: -

WebSphere Application Server and Liberty Security Update [PDF]

which was authored and presented at Impact in 2013 by Bill O'Donnell, STSM – WebSphere Foundation Security Architect IBM – WebSphere Development.

Bill also actively manages this important site: -

WebSphere Application Server security

Maintain and strengthen the security of your WebSphere Application Server environment

which is definitely worth book-marking.

MQ - AMQ9637: Channel is lacking a certificate

I wrote this post a wee while back, and those nice developerWorks folks have kindly published it for me: -

MQ - AMQ9637: Channel is lacking a certificate

...

In the context of configuring encrypted connections between WebSphere / IBM MQ and WebSphere Application Server, I'm currently building a scripted process to set up the infrastructure end-to-end, which I'm testing on a VM. For the record, I'm using IBM MQ 8.0.0.2 and WebSphere Application Server (WAS) 8.5.5.2.

...

At this point, I hit an issue. Instead of a happy message, I saw:

javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

More importantly, this is what I saw in the Queue Manager's log:

----- amqrmrsa.c : 930 --------------------------------------------------------
10/09/15 08:14:35 - Process(25255.10) User(mqm) Program(amqrmppa)
                    Host(bpm856.uk.ibm.com) Installation(Installation1)
                    VRMF(8.0.0.2) QMgr(TESTQM)
                   
AMQ9637: Channel is lacking a certificate.

EXPLANATION:
The channel is lacking a certificate to use for the SSL handshake. The channel
name is '????' (if '????' it is unknown at this stage in the SSL processing).

The remote host is '????'.

The channel did not start.
ACTION:
Make sure the appropriate certificates are correctly configured in the key
repositories for both ends of the channel.
...

Want to know what happened ?

Then read me on developerWorks here: -

MQ - AMQ9637: Channel is lacking a certificate

Liberty and DevOps, continuous delivery and deployment environment

This IBM® Redbooks® publication will provide an example approach for an agile IT team to implement DevOps capabilities into their software delivery of a Java application. Several tools will be introduced, that show how teams can achieve transparency, traceability and automation in their application life cycle with all the stakeholders, to deliver a high quality application that meets its initial requirements. The application being built will highlight the composable and dynamic nature of the Liberty runtime which allows developers to quickly get their applications up and running using only the programming model features required for their applications.

Target audience is IT developers, IT managers, IT architects, Project managers, Test manager, Test developers, Operations managers, Operations developers.

Table of contents

Chapter 1. Introduction
Chapter 2. Project Planning using Rational Team Concert and Rational Quality Manager
Chapter 3. Code Development, Source code management and Build
Chapter 4. Continuous Test
Chapter 5. Deployment
Chapter 6. Production Environment
Chapter 7. Conclusion
Appendix A. Additional material

Liberty and DevOps, continuous delivery and deployment environment

Integrate IBM MQ with your IBM Business Process Manager processes

Integrate IBM MQ with your IBM Business Process Manager processes

Learn the various approaches to send and receive messages from a business process through IBM® MQ, communicating with back-end systems. Walk through examples with sample code to see the steps that are required to integrate IBM MQ V8.x with IBM Business Process Manager (BPM) V8.5.x. Gain an understanding of fundamental approaches to use IBM MQ to exchange data with other systems from IBM BPM processes.

WebSphere Application Server - Variables and Databases

I saw this ongoing issue today: -

[07/10/15 20:34:33:714 BST] 0000008f ConnectionFac E   J2CA0009E: An exception occurred while trying to instantiate the ManagedConnectionFactory class com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl used by resource jdbc/Sample : java.lang.reflect.InvocationTargetException

[07/10/15 20:34:33:709 BST] 0000008f J2CUtilityCla E   J2CA0036E: An exception occurred while invoking method setDataSourceProperties on com.ibm.w
s.rsadapter.spi.WSManagedConnectionFactoryImpl used by resource jdbc/Sample : com.ibm.ws.exception.WsException: DSRA0023E: The DataSource implementation class "com.ibm.db2.jcc.DB2XADataSource" could not be found.

[07/10/15 20:34:33:723 BST] 0000008f SystemErr     R com.ibm.websphere.naming.CannotInstantiateObjectException: Exception occurred while the JNDI NamingManager was processing a javax.naming.Reference object. [Root exception is com.ibm.websphere.naming.CannotInstantiateObjectException: Exception occurred while the JNDI NamingManager was processing a javax.naming.Reference object. [Root exception is java.lang.reflect.InvocationTargetException]]

[07/10/15 20:34:33:728 BST] 0000008f SystemErr     R Caused by: com.ibm.websphere.naming.CannotInstantiateObjectException: Exception occurred while the JNDI NamingManager was processing a javax.naming.Reference object. [Root exception is java.lang.reflect.InvocationTargetException]

[07/10/15 20:34:33:734 BST] 0000008f SystemErr     R Caused by: com.ibm.ws.rsadapter.exceptions.DataStoreAdapterException: DSRA0023E: The DataSource implementation class "com.ibm.db2.jcc.DB2XADataSource" could not be found.

Caused by: java.lang.ClassNotFoundException: com.ibm.db2.jcc.DB2XADataSource

This came about because I was using a JDBC Provider and Data Source, both of which I'd created at the scope of the cell: -

Create JDBC Provider

cellID=AdminControl.getCell()

provider=AdminTask.createJDBCProvider('[-scope Cell='+cellID+' -databaseType DB2 -providerType "DB2 Using IBM JCC Driver" -implementationType "XA data source" -name "DB2 Using IBM JCC Driver (XA)" -description "Two-phase commit DB2 JCC provider that supports JDBC 4.0 using the IBM Data Server Driver for JDBC and SQLJ. IBM Data Server Driver is the next generation of the DB2 Universal JCC driver. Data sources created under this provider support the use of XA to perform 2-phase commit processing. Use of JDBC driver type 2 on WebSphere Application Server for Z/OS is not supported for data sources created under this provider. This provider is configurable in version 7.0 and later nodes." -classpath [${DB2_JCC_DRIVER_PATH}/db2jcc4.jar ${DB2_JCC_DRIVER_PATH}/db2jcc_license_cu.jar ] -nativePath [${DB2_JCC_DRIVER_PATH} ] ]')

Create JDBC Data Source

AdminTask.createDatasource(provider, '[-name Sample -jndiName jdbc/Sample -dataStoreHelperClassName com.ibm.websphere.rsadapter.DB2UniversalDataStoreHelper -containerManagedPersistence true -componentManagedAuthenticationAlias db2user1 -xaRecoveryAuthAlias -configureResourceProperties [[databaseName java.lang.String SAMPLE] [driverType java.lang.Integer 4] [serverName java.lang.String wasdemo.uk.ibm.com] [portNumber java.lang.Integer 60006]]]')

Whilst I was able to happily test the datasource: -

AdminControl.invoke('WebSphere:name=DataSourceCfgHelper,process=dmgr,platform=dynamicproxy,node=Dmgr,version=8.5.5.5,type=DataSourceCfgHelper,mbeanIdentifier=DataSourceCfgHelper,cell=WASCell,spec=1.0', 'testConnection', '[cells/WASCell|resources.xml#DataSource_1444136343751]', '[java.lang.String]')

my application failed with the above exceptions.

The solution ?

The application, running on the Application Server on the Node, is referencing a cell-scoped JDBC data source using a cell-scoped JDBC provider .... which has a WebSphere Variable - DB2_JCC_DRIVER_PATH - which is only available at the cell-level.

However, that wasn't the problem :-)

When I dug further, I *DID* have an identically named variable, scoped at the node-level: -

Can you see what I did wrong ?

Yes, I had a variable scoped at the node, which trumps the cell, which had a null value.

Once I fixed this: -

AdminConfig.modify('(cells/WASCell/nodes/AppSrv01Node|variables.xml#VariableSubstitutionEntry_1444248288880)', '[[symbolicName "DB2_JCC_DRIVER_PATH"] [description ""] [value "/opt/ibm/WebSphere/AppServer/jdbcdrivers/DB2/ "]]')

everything was lovely again :-)

This was definitely of use: -

J2CA0036E and DSRA0023E are logged in SystemOut.log when testing the connection or when attempting to create a connection from an application to a database.

as did this: -

Administrative console scope settings

Doh, WebSphere Application Server to DB2 via JDBC - Get the schema right, fool

I'm writing a tutorial for my team, taking them through the process of building a WebSphere Application Server (WAS) configuration, with IBM HTTP Server (IHS) and DB2.

For the WAS > DB2 piece, I've written the necessary Jython scripts to create a JDBC provider and data source, and deploy a sample Java EE web application.

However, during testing, I hit an interesting issue, evidenced by this exception: -

[07/10/15 19:41:21:224 BST] 0000008e SystemErr     R com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-204, SQLSTATE=42704, SQLERRMC=DB2USER1.EMPLOYEE, DRIVER=4.19.26

in SystemErr.log.

Long story short, I'm connecting from WAS to DB2 using the user ID db2user1 which has DBADM permissions to the database, the SAMPLE database that one creates using the command db2sampl.

However, when I ran the initial db2sampl command, I took the path of least resistance, aka the default :-)

Therefore, I'd created the data ( tables etc. ) in the wrong schema - db2inst1 - whereas WAS is expecting to use the schema db2user1

To prove this, I checked DB2: -

db2 connect to sample

   Database Connection Information

 Database server        = DB2/LINUXX8664 10.5.5
 SQL authorization ID   = DB2INST1
 Local database alias   = SAMPLE

db2 "select * from db2user1.employee"

SQL0204N  "DB2USER11.EMPLOYEE" is an undefined name.  SQLSTATE=42704

There are a number of solutions to this problem, I chose to drop and then recreate the SAMPLE database: -

drop database sample

db2sampl -schema db2user1

Now my query works: -

db2 "select * from db2user1.employee"

EMPNO  FIRSTNME     MIDINIT LASTNAME        WORKDEPT PHONENO HIREDATE   JOB      EDLEVEL SEX BIRTHDATE  SALARY      BONUS       COMM       
------ ------------ ------- --------------- -------- ------- ---------- -------- ------- --- ---------- ----------- ----------- -----------
000010 CHRISTINE    I       HAAS            A00      3978    01/01/1995 PRES          18 F   08/24/1963   152750.00     1000.00     4220.00
000020 MICHAEL      L       THOMPSON        B01      3476    10/10/2003 MANAGER       18 M   02/02/1978    94250.00      800.00     3300.00
000030 SALLY        A       KWAN            C01      4738    04/05/2005 MANAGER       20 F   05/11/1971    98250.00      800.00     3060.00
000050 JOHN         B       GEYER           E01      6789    08/17/1979 MANAGER       16 M   09/15/1955    80175.00      800.00     3214.00
000060 IRVING       F       STERN           D11      6423    09/14/2003 MANAGER       16 M   07/07/1975    72250.00      500.00     2580.00
000070 EVA          D       PULASKI         D21      7831    09/30/2005 MANAGER       16 F   05/26/2003    96170.00      700.00     2893.00
000090 EILEEN       W       HENDERSON       E11      5498    08/15/2000 MANAGER       16 F   05/15/1971    89750.00      600.00     2380.00
000100 THEODORE     Q       SPENSER         E21      0972    06/19/2000 MANAGER       14 M   12/18/1980    86150.00      500.00     2092.00
000110 VINCENZO     G       LUCCHESSI       A00      3490    05/16/1988 SALESREP      19 M   11/05/1959    66500.00      900.00     3720.00
000120 SEAN                 O'CONNELL       A00      2167    12/05/1993 CLERK         14 M   10/18/1972    49250.00      600.00     2340.00
000130 DELORES      M       QUINTANA        C01      4578    07/28/2001 ANALYST       16 F   09/15/1955    73800.00      500.00     1904.00
000140 HEATHER      A       NICHOLLS        C01      1793    12/15/2006 ANALYST       18 F   01/19/1976    68420.00      600.00     2274.00
000150 BRUCE                ADAMSON         D11      4510    02/12/2002 DESIGNER      16 M   05/17/1977    55280.00      500.00     2022.00
000160 ELIZABETH    R       PIANKA          D11      3782    10/11/2006 DESIGNER      17 F   04/12/1980    62250.00      400.00     1780.00
000170 MASATOSHI    J       YOSHIMURA       D11      2890    09/15/1999 DESIGNER      16 M   01/05/1981    44680.00      500.00     1974.00
000180 MARILYN      S       SCOUTTEN        D11      1682    07/07/2003 DESIGNER      17 F   02/21/1979    51340.00      500.00     1707.00
000190 JAMES        H       WALKER          D11      2986    07/26/2004 DESIGNER      16 M   06/25/1982    50450.00      400.00     1636.00
000200 DAVID                BROWN           D11      4501    03/03/2002 DESIGNER      16 M   05/29/1971    57740.00      600.00     2217.00
000210 WILLIAM      T       JONES           D11      0942    04/11/1998 DESIGNER      17 M   02/23/2003    68270.00      400.00     1462.00
000220 JENNIFER     K       LUTZ            D11      0672    08/29/1998 DESIGNER      18 F   03/19/1978    49840.00      600.00     2387.00
000230 JAMES        J       JEFFERSON       D21      2094    11/21/1996 CLERK         14 M   05/30/1980    42180.00      400.00     1774.00
000240 SALVATORE    M       MARINO          D21      3780    12/05/2004 CLERK         17 M   03/31/2002    48760.00      600.00     2301.00
000250 DANIEL       S       SMITH           D21      0961    10/30/1999 CLERK         15 M   11/12/1969    49180.00      400.00     1534.00
000260 SYBIL        P       JOHNSON         D21      8953    09/11/2005 CLERK         16 F   10/05/1976    47250.00      300.00     1380.00
000270 MARIA        L       PEREZ           D21      9001    09/30/2006 CLERK         15 F   05/26/2003    37380.00      500.00     2190.00
000280 ETHEL        R       SCHNEIDER       E11      8997    03/24/1997 OPERATOR      17 F   03/28/1976    36250.00      500.00     2100.00
000290 JOHN         R       PARKER          E11      4502    05/30/2006 OPERATOR      12 M   07/09/1985    35340.00      300.00     1227.00
000300 PHILIP       X       SMITH           E11      2095    06/19/2002 OPERATOR      14 M   10/27/1976    37750.00      400.00     1420.00
000310 MAUDE        F       SETRIGHT        E11      3332    09/12/1994 OPERATOR      12 F   04/21/1961    35900.00      300.00     1272.00
000320 RAMLAL       V       MEHTA           E21      9990    07/07/1995 FIELDREP      16 M   08/11/1962    39950.00      400.00     1596.00
000330 WING                 LEE             E21      2103    02/23/2006 FIELDREP      14 M   07/18/1971    45370.00      500.00     2030.00
000340 JASON        R       GOUNOT          E21      5698    05/05/1977 FIELDREP      16 M   05/17/1956    43840.00      500.00     1907.00
200010 DIAN         J       HEMMINGER       A00      3978    01/01/1995 SALESREP      18 F   08/14/1973    46500.00     1000.00     4220.00
200120 GREG                 ORLANDO         A00      2167    05/05/2002 CLERK         14 M   10/18/1972    39250.00      600.00     2340.00
200140 KIM          N       NATZ            C01      1793    12/15/2006 ANALYST       18 F   01/19/1976    68420.00      600.00     2274.00
200170 KIYOSHI              YAMAMOTO        D11      2890    09/15/2005 DESIGNER      16 M   01/05/1981    64680.00      500.00     1974.00
200220 REBA         K       JOHN            D11      0672    08/29/2005 DESIGNER      18 F   03/19/1978    69840.00      600.00     2387.00
200240 ROBERT       M       MONTEVERDE      D21      3780    12/05/2004 CLERK         17 M   03/31/1984    37760.00      600.00     2301.00
200280 EILEEN       R       SCHWARTZ        E11      8997    03/24/1997 OPERATOR      17 F   03/28/1966    46250.00      500.00     2100.00
200310 MICHELLE     F       SPRINGER        E11      3332    09/12/1994 OPERATOR      12 F   04/21/1961    35900.00      300.00     1272.00
200330 HELENA               WONG            E21      2103    02/23/2006 FIELDREP      14 F   07/18/1971    35370.00      500.00     2030.00
200340 ROY          R       ALONZO          E21      5698    07/05/1997 FIELDREP      16 M   05/17/1956    31840.00      500.00     1907.00

  42 record(s) selected.

I then saw: -

[07/10/15 20:11:05:359 BST] 0000009f SystemErr     R com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-551, SQLSTATE=42501, SQLERRMC=DB2USER1;SELECT;DB2USER1.EMPLOYEE, DRIVER=4.19.26

which I again validated in DB2 as follows: -

db2 connect to sample user db2user1 using passw0rd

   Database Connection Information

 Database server        = DB2/LINUXX8664 10.5.5
 SQL authorization ID   = DB2USER1
 Local database alias   = SAMPLE

db2 "select * from db2user1.employee"

SQL0551N  The statement failed because the authorization ID does not have the 
required authorization or privilege to perform the operation.  Authorization 
ID: "DB2USER1".  Operation: "SELECT". Object: "DB2USER1.EMPLOYEE".  
SQLSTATE=42501

Of course, I'd rebuilt the DB but forgot to grant db2user1 access to it :-)

This was easily fixed: -

db2 connect to sample user db2inst1 using passw0rd
db2 grant dbadm on database to user db2user1
db2 terminate

and re-tested: -

db2 connect to sample user db2user1 using passw0rd
db2 "select * from db2user1.employee"

...

000210 WILLIAM      T       JONES           D11      0942    11/04/1998 DESIGNER      17 M   23/02/2003    68270.00      400.00     1462.00
000220 JENNIFER     K       LUTZ            D11      0672    29/08/1998 DESIGNER      18 F   19/03/1978    49840.00      600.00     2387.00
000230 JAMES        J       JEFFERSON       D21      2094    21/11/1996 CLERK         14 M   30/05/1980    42180.00      400.00     1774.00
000240 SALVATORE    M       MARINO          D21      3780    05/12/2004 CLERK         17 M   31/03/2002    48760.00      600.00     2301.00
000250 DANIEL       S       SMITH           D21      0961    30/10/1999 CLERK         15 M   12/11/1969    49180.00      400.00     1534.00
000260 SYBIL        P       JOHNSON         D21      8953    11/09/2005 CLERK         16 F   05/10/1976    47250.00      300.00     1380.00
000270 MARIA        L       PEREZ           D21      9001    30/09/2006 CLERK         15 F   26/05/2003    37380.00      500.00     2190.00
000280 ETHEL        R       SCHNEIDER       E11      8997    24/03/1997 OPERATOR      17 F   28/03/1976    36250.00      500.00     2100.00
000290 JOHN         R       PARKER          E11      4502    30/05/2006 OPERATOR      12 M   09/07/1985    35340.00      300.00     1227.00
...

and, more importantly, my Java code now works :-)

Which is nice :-)