Monday, 29 June 2015

IBM Business Process Manager concepts and terminology

Some useful background about IBM Business Process Manager .. from the IBM Integration Bus documentation: -

IBM Business Process Manager concepts and terminology

You can become more familiar with Business Process Manager (BPM) concepts to clarify how it interacts with IBM® Integration Bus.

What is Process Designer?

IBM Process Designer is a graphical user interface tool. You can use it to quickly model and implement your business processes. You then test and demonstrate the process design during the development phase. For more information about Process Designer, look for getting started with Process Designer in the Business Process Manager product documentation.

What is a Process Center?

You use the tools and repository of a Process Center to model the processes, services, and other assets in a business process by using IBM Process Designer. For more information about Process Center, look for process designer and process center tasks in the Business Process Manager product documentation.

What are business process definitions?

A business process definition is a model, you create in Process Designer, of a business process. For more information, look for business process definitions (BPDs) in the Business Process Manager product documentation.

What are business objects?

A business object is a set of fields or elements that you use together to represent a meaningful business process. You define business objects as variables to pass information through a business process. Each business object can be a primitive data type (such as, a string or an integer) or it can itself be a business object. For more information, look for business objects in the Business Process Manager product documentation.

What are system tasks and Integration Services?

System tasks are activities in a business process definition that are completed by an automated system. For more information, look for understanding task types in the BPM information centre.

An Integration Service is a particular type of system task that you use to access the web and other services. For more information, look for understanding service types in the Business Process Manager product documentation.

What are toolkits?

A BPM toolkit is a library of process designer artifacts; such as business objects, Integration Services, or business process definitions. A toolkit can be used for sharing between business process definitions. Toolkits are stored and managed with Process Center. For more information about toolkits, look for managing toolkits in the Business Process Manager product documentation.

What are snapshots?

You make versions by using snapshots. A snapshot records the state of library items (such as toolkits) at a specific time. For more information, look for creating snapshots in the process center console in the Business Process Manager product documentation.

What is a BPM export .twx file?

You import an export .twx file to transport BPM artifacts between Process Centers or to integrate with IBM Integration Bus. The format for the export .twx file is proprietary. For more information, look for importing and exporting process applications from the process center console in the Business Process Manager product documentation.

Sunday, 28 June 2015

Whilst I remember - parsing my JSON in IBM Integration Bus

Following my earlier post: -

I'm writing this down before I forget.

Previously,I'd noticed that my Compute Node was returning JSON in this format: -

message <{"Employee":{"row":{"EMPNO":"000100","FIRSTNME":"THEODORE","LASTNAME":"SPENSER"}}}>

which didn't really tie up with my expectations, given that the DB2 schema is like this: -


I fiddled around with my ESQL, changing it from this: -

SET OutputRoot.JSON.Data.Employee = rowRef;

to: -

SET OutputRoot.JSON.Data.Employee = dbResultSetRef.row;

In other words, I removed the reference to rowRef and instead replaced it with dbResultSetRef.row.

I now see this: -

message <{"Employee":{"EMPNO":"000100","FIRSTNME":"THEODORE","LASTNAME":"SPENSER"}}>
message <{"Employee":{"EMPNO":"000200","FIRSTNME":"DAVID","LASTNAME":"BROWN"}}>
message <{"Employee":{"EMPNO":"000300","FIRSTNME":"PHILIP","LASTNAME":"SMITH"}}>

from input such as this: -


I also learned about the Trace Node

which returns some tasty debug: -

Jun 28 20:37:54 bpmdemo IIB[77236]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 90570] (Msg 1/1) BIP3051E: Error message '( ['MQROOT' : 0x7f68b80d3e00]   (0x01000000:Name  ):Properties = ( ['MQPROPERTYPARSER' : 0x7f68b803c390]     (0x03000000:NameValue):MessageSet             = NULL     (0x03000000:NameValue):MessageType            = NULL     (0x03000000:NameValue):MessageFormat          = NULL     (0x03000000:NameValue):Encoding               = NULL     (0x03000000:NameValue):CodedCharSetId         = NULL     (0x03000000:NameValue):Transactional          = NULL     (0x03000000:NameValue):Persistence            = NULL     (0x03000000:NameValue):CreationTime           = NULL     (0x03000000:NameValue):ExpirationTime         = NULL     (0x03000000:NameValue):Priority               = NULL     (0x03000000:NameValue):ReplyIdentifier        = NULL     (0x03000000:NameValue):ReplyProtocol          = 'MQ' (CHARACTER)     (0x03000000:NameValue):Topic                  = NULL     (0x03000000:NameValue):ContentType            = NULL     (0x03000000:NameValue):IdentitySourceType     = NULL     (0x03000000:NameValue):IdentitySourceToken    = NULL     (0x03000000:NameValue):IdentitySourcePassword = NULL     (0x03000000:NameValue):IdentitySourceIssuedBy = NULL     (0x03000000:NameValue):IdentityMappedType     = NULL     (0x03000000:NameValue):IdentityMappedToken    = NULL     (0x03000000:NameValue):IdentityMappedPassword = NULL     (0x03000000:NameValue):IdentityMappedIssuedBy = NULL   )   (0x01000000:Object):JSON       = ( ['json' : 0x7f68b81666a0]     (0x01000000:Object):Data = (       (0x01000000:Object):Employee = (         (0x03000000:NameValue):EMPNO    = '000200' (CHARACTER)         (0x03000000:NameValue):FIRSTNME = 'DAVID' (CHARACTER)         (0x03000000:NameValue):LASTNAME = 'BROWN' (CHARACTER)       )     )   ) ) ' from trace node 'employeeService.Trace'. 
Jun 28 20:38:08 bpmdemo IIB[77236]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 90570] (Msg 1/1) BIP3051E: Error message '( ['MQROOT' : 0x7f68b80d3e00]   (0x01000000:Name  ):Properties = ( ['MQPROPERTYPARSER' : 0x7f68b803c390]     (0x03000000:NameValue):MessageSet             = NULL     (0x03000000:NameValue):MessageType            = NULL     (0x03000000:NameValue):MessageFormat          = NULL     (0x03000000:NameValue):Encoding               = NULL     (0x03000000:NameValue):CodedCharSetId         = NULL     (0x03000000:NameValue):Transactional          = NULL     (0x03000000:NameValue):Persistence            = NULL     (0x03000000:NameValue):CreationTime           = NULL     (0x03000000:NameValue):ExpirationTime         = NULL     (0x03000000:NameValue):Priority               = NULL     (0x03000000:NameValue):ReplyIdentifier        = NULL     (0x03000000:NameValue):ReplyProtocol          = 'MQ' (CHARACTER)     (0x03000000:NameValue):Topic                  = NULL     (0x03000000:NameValue):ContentType            = NULL     (0x03000000:NameValue):IdentitySourceType     = NULL     (0x03000000:NameValue):IdentitySourceToken    = NULL     (0x03000000:NameValue):IdentitySourcePassword = NULL     (0x03000000:NameValue):IdentitySourceIssuedBy = NULL     (0x03000000:NameValue):IdentityMappedType     = NULL     (0x03000000:NameValue):IdentityMappedToken    = NULL     (0x03000000:NameValue):IdentityMappedPassword = NULL     (0x03000000:NameValue):IdentityMappedIssuedBy = NULL   )   (0x01000000:Object):JSON       = ( ['json' : 0x7f68b81666a0]     (0x01000000:Object):Data = (       (0x01000000:Object):Employee = (         (0x03000000:NameValue):EMPNO    = '000300' (CHARACTER)         (0x03000000:NameValue):FIRSTNME = 'PHILIP' (CHARACTER)         (0x03000000:NameValue):LASTNAME = 'SMITH' (CHARACTER)       )     )   ) ) ' from trace node 'employeeService.Trace'. 

Event Sequencing Service and IBM BPM Advanced

I was seeing this :-

[6/28/15 7:26:09:973 BST] 00000379 EsMeConnectio W run() CWRES0053W: The event sequencing service cannot connect to the messaging engine on the Service Component Architecture (SCA) system bus that hosts the destination sca/demo101/component/sendID.

whilst testing a BPEL component in IBM Integration Designer 8.5.6, against an IBM BPM Advanced runtime.

This kinda came out of nowhere, but the key phrase event sequencing rang a bell.

I'd been fiddling around with the properties of my BPEL ( prior to realising that I'd not changed it from the default Long-Running Process to Microflow.

I'd specifically changed one of the Qualities of Service (QoS) parameters for one of my interfaces: -

specifically this one: -

In other words, I'd enabled Event Sequencing on the runtime, which was something that I did not specifically need.

Once I removed the Qualifier, and restarted the Process Server runtime, life was good again.

Saturday, 27 June 2015

IBM Business Process Manager and Desktop SSO via Kerberos and SPNEGO


This comes on the back of a similar article describing how one can configure IBM Operational Decision Manager (ODM) 8.6 to work with Microsoft Active Directory 2008, and deliver desktop Single Sign-On for Windows users.

In this scenario, ODM has been "replaced" by IBM BPM Advanced on IBM WebSphere Application Server (WAS)

Again, the objective is to configure WAS to use Active Directory 2008 for identity management, authentication and authorisation ( access control ). In addition, Kerberos / SPNEGO will be configured in AD and in WAS, in order to allow Windows desktop users to automatically access BPM functionality without being challenged for credentials ( using their existing Windows log in ).

This article is broken into sections, and requires each section to be completed in order. In addition, it is assumed that AD has already been configured, and that the relevant users and groups have been created.

The following is intended to provide an example of the typical users and groups that one might require: -

WAS > AD Bind User

dn: CN=ldapbind,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

WAS Administration User

dn: CN=WebSphere Admin,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

WAS Administration Group

dn: CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com
member: CN=WebSphere Admin,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

BPM Administration Group

dn: CN=BPMAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com
member: CN=Administrator,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

BPM Authors Group

dn: CN=BPMAuthors,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com
member: CN=Administrator,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com

BPM Users Group

dn: CN=BPMUsers,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com
member: CN=Administrator,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com


  • IBM BPM Advanced installed on WebSphere Application Server Network Deployment, with IBM HTTP Server and IBM WebSphere Plugin.
  • IHS/Plugin federated into WAS cell
  • IHS configured for SSL
  • WAS Virtual Host for IHS SSL port added ( 8443 in this case )
  • IHS-hosted SSL certificate in WAS cell-default trust store

WAS Tasks

Start WSAdmin

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/ -lang jython

Create LDAP Repository

AdminTask.createIdMgrLDAPRepository('[-default true -id -adapterClassName -ldapServerType AD -sslConfiguration -certificateMapMode exactdn -supportChangeLog none -certificateFilter -loginProperties uid]')

Add AD Server

AdminTask.addIdMgrLDAPServer('[-id -host -bindDN CN=ldapbind,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com -bindPassword Qk525mtp -referal ignore -sslEnabled false -ldapServerType AD -sslConfiguration -certificateMapMode exactdn -certificateFilter -authentication simple -port 389]')

Add Base Entry

AdminTask.addIdMgrRepositoryBaseEntry('[-id -name CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com -nameInRepository CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com]')

Add Realm Base Entry

AdminTask.addIdMgrRealmBaseEntry('[-name defaultWIMFileBasedRealm -baseEntry CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com]')

Add User Object Classes and Search Base

AdminTask.updateIdMgrLDAPEntityType('[-id -name PersonAccount -objectClasses user;organizationalPerson;person -searchBases CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com -searchFilter (ObjectCategory=User)]')

Add Group Object Class and Search Base

AdminTask.updateIdMgrLDAPEntityType('[-id -name Group -objectClasses group -searchBases CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com -searchFilter (ObjectCategory=Group)]')

Enable Allow Operation If Repository Down

AdminTask.updateIdMgrRealm('[-name defaultWIMFileBasedRealm -allowOperationIfReposDown true]')

Add AD Group ( ) into Admin Roles

AdminTask.mapGroupsToAdminRole('[-roleName adminsecuritymanager -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName administrator -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAuditRole('[-roleName auditor -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName configurator -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName deployer -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName iscadmins -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName monitor -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')
AdminTask.mapGroupsToAdminRole('[-roleName operator -accessids [group:defaultWIMFileBasedRealm/CN=WebSphereAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ] -groupids [WebSphereAdmins@defaultWIMFileBasedRealm ]]')

Save and Sync

AD Tasks

Create Service Principal Name

setspn -a HTTP/ ldapbind

Validate SPN

setspn -l ldapbind

Create Kerberos Keytab

ktpass -out bpm855.keytab -princ HTTP/ -mapUser AD2008\ldapbind -mapOp set -pass Qk525mtp -ptype KRB5_NT_PRINCIPAL

WAS Tasks

Validate Kerberos Keytab


ktutil: rkt bpm855.keytab
ktutil: list

slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    3            HTTP/

ktutil: quit

/opt/IBM/WebSphere/AppServer/java/jre/bin/java -e -k /home/wasadmin/bpm855.keytab

Key table: /home/wasadmin/bpm855.keytab
Number of entries: 1

[1] principal: HTTP/

Encryption type: RC4 with HMAC

Create Kerberos Configuration File

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/ -lang jython

AdminTask.createKrbConfigFile("[-krbPath /opt/IBM/WebSphere/AppServer/java/jre/lib/security/krb5.conf -realm UK.IBM.COM -kdcHost -dns -keytabPath /home/wasadmin/bpm855.keytab -encryption des3-cbc-sha1]")

'/opt/IBM/WebSphere/AppServer/java/jre/lib/security/krb5.conf has been created.'


cat /opt/IBM/WebSphere/AppServer/java/jre/lib/security/krb5.conf

   default_realm = UK.IBM.COM
   default_keytab_name = FILE:/home/
   default_tkt_enctypes = des3-cbc-sha1
   default_tgs_enctypes = des3-cbc-sha1
   forwardable  = true
   renewable  = true
   noaddresses = true
   clockskew  = 300
   UK.IBM.COM = {
     kdc =
     default_domain =

Add SPNEGO Configuration

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/ -lang jython
AdminTask.addSpnegoFilter('[-hostName -krb5Realm AD2008 -filterCriteria -filterClass -spnegoNotSupportedPage -ntlmTokenReceivedPage -trimUserName true -enabledGssCredDelegate false]')
AdminTask.configureSpnego('[-enabled true -dynamicReload true -allowAppAuthMethodFallback true -krb5Config /opt/IBM/WebSphere/AppServer/java/jre/lib/security/krb5.conf -krb5Keytab /home/wasadmin/bpm855.keytab]')
AdminTask.setAdminActiveSecuritySettings('[-customProperties [""] ]')

Create Fallback Login Page - /opt/IBM/HTTPServer/htdocs/NoSpnegoRedirect.html

This page is used when (a) one uses a browser that is not configured for SPNEGO e.g. Chrome or Safari etc. and (b) when one chooses NOT to invoke SSO by adding ?noSPNEGO to the URL

<!DOCTYPE HTML PUBLIC "-//W3C/DTD HTML 4.0 Transitional//EN">
<META HTTP-EQUIV="Content-Type" CONTENT="text/html">
- This file should be served from an unprotected website. Alternatively, it can be loaded from the WebSphere Application Server file system.
- Any imbedded graphics/javascript/css must be loaded from an unprotected website.
- This file is loaded after WebSphere Application Server is initialized. If changes to this file are necessary, restart WebSphere Application Server.
- This file is returned whenever the SPNEGO TAI receives an NTLM token for any application in the cell. In other words, this file is generic for all applications. However, by using the  document.location Javascipt,
- we can get the original URL, and redirect to that original URL with the "?noSPNEGO" text added - thus forcing the standard application userid/password challenge.
<script language="javascript">
var origUrl=""+document.location;
   if (origUrl.indexOf("noSPNEGO")<0) {
if (origUrl.indexOf('?')>=0) origUrl+="&noSPNEGO";
else origUrl+="?noSPNEGO";
function redirTimer() {

<META HTTP-EQUIV = "Pragma" CONTENT="no-cache">
<script language="javascript">
document.write("<title> Redirect to "+origUrl+ " </title>");
<body onLoad="redirTimer()"/>

Add Fallback Login Page to WAS

This also updates the SPNEGO filters to disable SSO between Process Center and Process Server - see SPNEGO authentication might cause communication issues between the Process Server and the Process Center for IBM Business Process Manager (BPM)

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/ -lang jython
AdminTask.modifySpnegoFilter('[-hostName -krb5Realm AD2008 -filterCriteria request-url!=ProcessCenterInternal;request-url!=ProcessServerInternal;request-url^=ProcessCenter|ProcessAdmin|portal|ProcessPortal|teamworks|PerformanceAdmin;request-url!=noSPNEGO; -filterClass -spnegoNotSupportedPage -ntlmTokenReceivedPage -trimUserName true -enabledGssCredDelegate false]')

Restart Deployment Environment

/opt/IBM/WebSphere/AppServer/bin/ -stop -profile Dmgr01 -de De1

/opt/IBM/WebSphere/AppServer/bin/ -start -profile Dmgr01 -de De1

Windows Desktop Tasks ( IE )

Enable Integrated Windows Authentication (IWA), if not already enabled: -


Add to Local Intranet Trusted Zone: -

Restart IE and access Process Center URL: -

SSO should now work i.e. one should be logged into PC as the Windows user - Administrator: -

The same should work for Process Admin: -

and Process Portal: -

By default, however, SSO for Performance Admin: -

will not work, purely because the only user authorised to access Performance Admin is deAdmin.

This means that the login page will be presented, with a corresponding message in SystemOut.log ( for the SupportCluster ): -

[14/11/14 10:17:18:401 GMT] 00000143 WebCollaborat A   SECJ0129E: Authorization failed for user Administrator:defaultWIMFileBasedRealm while invoking GET on default_host:/PerformanceAdmin/console/, Authorization failed, Not granted any of the required roles: twuser

If needed, this could be mitigated by adding, for example, BPMAdmins into the twuser role for the Performance Admin application: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/ -lang jython
AdminApp.edit('IBM_BPM_PerformanceDW_SupCluster', '[ -MapRolesToUsers [[ twuser AppDeploymentOption.No AppDeploymentOption.No deAdmin BPMAdmins AppDeploymentOption.No " " group:defaultWIMFileBasedRealm/CN=BPMAdmins,CN=Users,DC=ad2008,DC=uk,DC=ibm,DC=com ]]]' )

However, this requires insight from the business, as the security requirements for the Performance Admin console may differ to those for Process Center, Process Portal, Process Admin etc.

Optional and Additional Steps

BPM internal groups can be updated using Process Admin in order to enforce specific roles when using SSO: -

Internal BPM Group AD Group

tw_authors BPMAuthors
tw_admins BPMAdmins
tw_allusers BPMUsers

Success in Pictures

Process Center

Process Portal

Process Admin

Performance Admin


Thursday, 25 June 2015

WAS and Kerberos and SPNEGO - it's been a while - but it's all right now, in fact it it's a gas

It's been a while since first I worked with WAS, Kerberos and SPNEGO - 2011 was SUCH a long time ago.

We're configuring desktop SSO between WAS ND ( specifically for BPM Advanced but that's not important right now ) and MS Active Directory 2008.

I'd previously documented the required steps to create the Service Principal Name (SPN) and Kerberos key tab in AD, and I was confident that there were no issues there.

However, when we started up the WAS JVM ( we were specifically looking at the Deployment Manager ), we saw the following in SystemOut.log: -

[25/06/15 14:05:19:001 BST] 0000015a ServerCredent E initialize CWSPN0014E: An exception occurred during Kerberos initialization. Failure: org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS.

[25/06/15 14:05:19:008 BST] 0000015a ServerCredent E initializeServer CWSPN0015E: Unable to create a GSSCredential for: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:008 BST] 0000015a ServerCredent E initializeServerCreds CWSPN0017E: Unable to create GSSCredentials for any of the hosts specified in the configuration properties.

[25/06/15 14:05:19:048 BST] 0000015a TrustAssociat E initialize CWSPN0009E: SPNEGO Trust Association Interceptor configuration is not valid. Failure condition: initializeServerCreds: no credentials successfully created.. If you are not using the SPNEGO TAI, you can ignore this message.

and, in the FFDC logs: -

org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials
minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:005 BST]     FFDC Exception:org.ietf.jgss.GSSException ProbeId:68 org.ietf.jgss.GSSException, major code: 13, minor code: 0 major string: Invalid credentials
minor string: Cannot get credential from JAAS Subject for principal: HTTP/bpm101@2008ADMINLDAPS

[25/06/15 14:05:19:023 BST]     FFDC ProbeId:332 initializeServerCreds: no credentials successfully created.

We started by checking the WAS configuration and then moved on to the Kerberos key tab: -

/opt/ibm/WebSphere/AppServer/java/jre/bin/java -e -k /home/wasadmin/keytabs/bpm.keytab

which returns: -

Key table: /home/wasadmin/keytabs/bpm.keytab
Number of entries: 1

1] principal: HTTP/
        KVNO: 4

       Encryption type: RC4 with HMAC

We also checked the WAS Kerberos configuration file, krb5.conf.

        default_realm =
        default_keytab_name = FILE:/home/wasadmin/keytabs/bpm.keytab
        default_tkt_enctypes = des3-cbc-sha1
        default_tgs_enctypes = des3-cbc-sha1
        forwardable  = true
        renewable  = true
        noaddresses = true
        clockskew  = 300

[realms] = {
                kdc =
                default_domain =

[domain_realm] =

We then started to look at the hostname of the server, as compared with what DNS shows.

Looking back at an older blog post ( from June 2014 ): -

I compiled and tested a Java class: -


public class hostStuff
public static void main(String[] args)

InetAddress address = InetAddress.getLocalHost();
System.out.println("My IP address ( via InetAddress.getLocalHost() ) is " + address.toString());
System.out.println("My hostname ( via InetAddress.getHostName() ) is " + address.getHostName());
System.out.println("My hostname ( via InetAddress.getCanonicalHostname() ) is  " + address.getCanonicalHostName());
catch (UnknownHostException e)
       System.out.println("I'm sorry. I don't know my own name.");

which showed that, whilst we were seeing the correct hostname, we were ONLY seeing the short name, rather than the fully-qualified service name.

In other words, we were seeing bpm101 rather than

When we checked further, that's what /etc/hosts was configured to use.

We asked the AIX sysadmin to change this - he commented out the entry in the hosts file - and then restarted WAS.

Immediately, SPNEGO burst into life.

Bottom line, it doesn't matter how good the SPNEGO and WAS configuration is, it is CRUCIAL to ensure that the hostname that WAS can see, via the various Java methods, matches what's been configured for the Service Principal Name (SPN).

Life is, as they say, a learning curve :-)

Oh Dear Busy Crying - Where's my ODBC data source ?

This is definitely a "Slap myself" moment.

Running my IBM Integration Bus (IIB) flow, the one that goes to DB2 via ODBC, produced this in /var/log/user.log: -

Jun 24 18:39:30 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 1/4) BIP2628E: Exception condition detected on input node 'customerService.Input'. 
Jun 24 18:39:30 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 2/4) BIP2230E: Error detected whilst processing a message in node 'customerService.Compute'. 
Jun 24 18:39:30 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 3/4) BIP2393E: Database error: ODBC return code '-1' from data source 'SAMPLE' using ODBC driver manager ''. 
Jun 24 18:39:30 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 4/4) BIP2347E: Database error: SQL State 'IM002'; Native Error Code '0'; Error Text 'System information file not found. Please check the ODBCINI and the ODBCSYSINI environment variables are correctly set.' 
Jun 24 18:39:31 bpmdemo IIB[4439]: IBM Integration Bus v9002 (IB9NODE.IIB9) [Thread 4553] (Msg 1/1) BIP2648E: Message backed out to a queue; node 'customerService.Input'. 

The reason was obvious to me, even though I'd forgotten to do it :-)

When I start IIB, I need to ensure that a pair of variables are set: -

export ODBCINI=/etc/odbc.ini 
export ODBCSYSINI=/etc

or the Bus cannot "see" the ODBC data source.

Short term, I scripted the startup of IIB thusly: -

#! /bin/bash
export ODBCINI=/etc/odbc.ini 
export ODBCSYSINI=/etc
mqsistart IB9NODE

but I could've easily set them in the owning user's ( wmbadmin ) startup script - .bashrc or .profile.

Which is nice :-)

Getting Started with WebSphere MQ Message Bindings in WebSphere Process Server and WebSphere Enterprise Service Bus

A relatively oldie ( 2011 ) but a goodie: -

This WebSphere Support Technical Exchange is designed to show you how to get the WebSphere MQ and JMS Messaging Bindings to work with WebSphere Enterprise Service Bus (WESB), including how to test them to demonstrate that they are working correctly.

This is a nice tutorial showing one how to setup MQ and create a simple SCA module. Although this was written for WebSphere Integration Developer ( now IBM Integration Designer ) and WebSphere Process Server ( now IBM Business Process Manager ), the core content is still extremely relevant.

Go on, check it out, you know you want to

Tuesday, 23 June 2015

Ouch, where's my disk space gone ?

I've got a development VM that hosts a whole bunch of software including: -

IBM WebSphere MQ
IBM Integration Bus
IBM Integration Designer
IBM Integration Toolkit
IBM WebSphere MQ Explroer
IBM Business Process Manager Advanced

The VM has a 80 GB disk, and yet .... quelle horreur ... I saw an exception relating to insufficient disk space: -

An exception occurred while writing to the platform log: No space left on device

Firstly I checked using df ( Display Freespace ): -

Filesystem            Size  Used Avail Use% Mounted on
                       75G   70G  589M 100% /
tmpfs                 3.9G   20K  3.9G   1% /dev/shm
/dev/sda1             477M   34M  419M   8% /boot

which doesn't look good.

Then, following this post: -

I tried this: -

sudo du -h / | grep -P '^[0-9\.]+G'

which returned: -

2.8G /opt/IBM/WebSphere/AppServer/cognos
1.1G /opt/IBM/WebSphere/AppServer/BPM/Lombardi
1.3G /opt/IBM/WebSphere/AppServer/BPM
7.9G /opt/IBM/WebSphere/AppServer
8.1G /opt/IBM/WebSphere
2.2G /opt/IBM/IMShared/plugins
1.7G /opt/IBM/IMShared/files
4.7G /opt/IBM/IMShared
15G /opt/IBM
1.6G /opt/mqm
1.1G /opt/ibm/db2/V10.5
1.1G /opt/ibm/db2
2.1G /opt/ibm
19G /opt
1.1G /home/wasadmin
2.2G /home/db2inst1/db2inst1/NODE0000/BPMDB/T0000002
2.5G /home/db2inst1/db2inst1/NODE0000/BPMDB
1.5G /home/db2inst1/db2inst1/NODE0000/CMNDB/T0000002
1.7G /home/db2inst1/db2inst1/NODE0000/CMNDB
7.2G /home/db2inst1/db2inst1/NODE0000
7.2G /home/db2inst1/db2inst1
7.4G /home/db2inst1
8.5G /home
du: cannot read directory `/mnt/hgfs/Software': No such file or directory
du: cannot access `/proc/36184/task/36184/fd/4': No such file or directory
du: cannot access `/proc/36184/task/36184/fdinfo/4': No such file or directory
du: cannot access `/proc/36184/fd/4': No such file or directory
du: cannot access `/proc/36184/fdinfo/4': No such file or directory
1.9G /usr
3.3G /var/repo/rhel66
3.3G /var/repo
38G /var/mqm/trace
38G /var/mqm
41G /var

70G /

which was very revealing.

More than half of my available disk was being used by ... WebSphere MQ.

When I checked more closely, I could definitely see a few likely candidates in /var/mqm/trace including: -

-rw-rw-r--   1 mqm mqm 266M Jun 23 12:08 mqjms_25962.trc
-rw-rw-r--   1 mqm mqm 8.2G Jun 23 14:15 mqjms_27216.trc
-rw-rw-r--   1 mqm mqm  21G Jun 21 21:02 mqjms_32695.trc

I shut down my Queue Manager: -

endmqm IB9QMGR -c

Quiesce request accepted. The queue manager will stop when all outstanding work
is complete.

and checked that it was stopped: -


QMNAME(IB9QMGR)                                           STATUS(Ended immediately)

and .... deleted the trace files: -

rm -Rf /var/mqm/trace

which brings things back to usual: -

df -kmh

Filesystem            Size  Used Avail Use% Mounted on
                       75G   33G   38G  47% /
tmpfs                 3.9G   20K  3.9G   1% /dev/shm
/dev/sda1             477M   34M  419M   8% /boot

Phew :-)

Monday, 22 June 2015

No space at my table - IBM BPM and

I saw this earlier, whilst starting a Process Server instance, specifically the Integration Test Environment within IBM Integration Designer (IID): -

[22/06/15 14:37:33:359 BST] 00000069 SibMessage    E   [BPM.ProcessServer.Bus:Node1.server1-BPM.ProcessServer.Bus] CWSIS0002E: The messaging engine encountered an exception while starting. Exception: CWSIS1501E: The data source has produced an unexpected exception: DB2 SQL Error: SQLCODE=-1585, SQLSTATE=54048, SQLERRMC=null, DRIVER=4.11.69

I suspected that the root cause MAY have been insufficient disk space within the Windows VM.

However, clearing down disk space and restarting BPM didn't help.

Now, obviously I didn't think to restart DB2 ......

However, I did find this Technote: -

which, whilst NOT relating to IBM BPM etc., did say: -


The DB2 instance did not have all 4 sizes of Temp tablespace defined.

Resolving the problem

To resolve the issue, the DB2 instance must have all 4 sizes of Temp tablespace created: 4k, 8k, 16k, and 32k. In addition, these must be set as System Temp tablespaces, and not as User Temp tablespaces.

I did create a System Temporary Tablespace: -

db2 create system temporary tablespace tsp32k pagesize 32K

 and BPM started happily.

Next time around, I'll do some more digging in DB2 land, as the BPM Process Server build doesn't specifically request a TS of this nature/size.

Which is nice :-)

Saturday, 20 June 2015

JMSFMQ6312 - Fun and more games with WebSphere Application Server, JMS and WebSphere MQ

A long time ago, in an office far far away, I hit a problem with WebSphere MQ and Java Messaging Service and WebSphere Application Server.

Specifically, the issue related to the interaction between WAS and WebSphere MQ using the Messaging Providers built into WAS.

I wrote about it here: -

This pertained to WAS on Unix.

This time around, I'm hitting the same issue, with WAS on Windows, as part of an IBM Integration Designer 8.5.6 deployment.

This is what I saw in the WAS server's SystemOut.log: -

[20/06/15 05:51:15:435 BST] 00000216 SibMessage    W   [:] CWSJY0003W: MQJCA4023: Startup reconnection failed for ActivationSpec 'javax.jms.Queue:CustomerService/MQJmsImport1_MQ_RECEIVE_D@IB9QMGR <-1336863013>'. Exception details: '
                       Message : JMSFMQ6312: An exception occurred in the Java(tm) MQI.
The Java(tm) MQI has thrown an exception describing the problem. 
See the linked exception for further information.
                         Class : class

 | Failed to load the WebSphere MQ native JNI library: 'mqjbnd'. 
 | The JVM attempted to load the platform native library 'mqjbnd', 
 | which was mapped to the filename: 'mqjbnd.dll'. 
 | When attempting to load the library, the JVM reported the error 
 | message: 
 | 'mqjbnd (c:\Program Files\IBM\WebSphere MQ\java\lib\mqjbnd.dll is not a valid Win32 application. )' 
 | The JVM's bit-size is: '64' 
 | The library path which was used to locate this library was: 
 | '*** Configured java.library.path **********************************
|  "C:\IBM\IID\PS\v8.5/lib/native/win/x86_64/"
|  "C:\IBM\IID\PS\v8.5\java_1.7_64\jre\bin\compressedrefs"
|  "C:\IBM\IID\PS\v8.5\java_1.7_64\jre\bin"
|  "C:\Windows\system32"
|  "C:\Windows"
|  "C:\IBM\IID\PS\v8.5\lib\native\win\x86_64"
|  "C:\IBM\IID\PS\v8.5\bin"
|  "C:\IBM\IID\PS\v8.5\java_1.7_64\bin"
|  "C:\IBM\IID\PS\v8.5\java_1.7_64\jre\bin"
|  "c:\Program Files\IBM\WebSphere MQ\java\lib"
|  "c:\Program Files\IBM\WebSphere MQ\java\lib64"
|  "C:\Windows\system32"
|  "C:\Windows"
|  "C:\Windows\system32\wbem"
|  "C:\Windows\system32\windowspowershell\v1.0\"
|  "c:\program files\ibm\gsk8\lib64"
|  "c:\program files (x86)\ibm\gsk8\lib"
|  "C:\IBM\IID\PS\v8.5\db2\BIN"
|  "C:\IBM\IID\PS\v8.5\db2\FUNCTION"
|  "C:\IBM\IID\PS\v8.5\db2\SAMPLES\REPL"
|  "c:\Program Files\IBM\WebSphere MQ\bin64"
|  "c:\Program Files\IBM\WebSphere MQ\bin"
|  "c:\Program Files\IBM\WebSphere MQ\tools\c\samples\bin"
|  "."
|  ""
| ********************************************************************' 
 | Check that the bit size of the JVM matches the bit size of the first 
 | native library file located within this java.library.path directory 
 | list. 
 | The native library 'mqjbnd' is used by the WebSphere MQ classes for 
 | Java and WebSphere MQ classes for JMS when creating a connection to 
 | the queue manager using a 'bindings' mode connection. A bindings 
 | mode connection is a connection which uses the system's memory to 
 | communicate with the queue manager, as opposed to a 'client' mode 
 | connection which uses a TCP/IP socket. 
 | In order to communicate with a queue manager using a bindings mode 
 | connection, the queue manager must be located on the same system as 
 | the WebSphere MQ classes for Java/JMS. If this is not the case in 
 | your environment, consider reconfiguring the application to utilise 
 | client mode connections. 

     Caused by [2] --> Message : java.lang.UnsatisfiedLinkError: mqjbnd (c:\Program Files\IBM\WebSphere MQ\java\lib\mqjbnd.dll is not a valid Win32 application. )
[20/06/15 05:51:15:439 BST] 00000216 ActivationSpe E   J2CA0138E: The Message Endpoint activation failed for ActivationSpec CustomerService/MQJmsImport1_AS ( and MDB application CustomerServiceApp#CustomerServiceMQJmsImport1#CustomerService due to the following exception: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.

The solution ?

Yes, I needed to set the Native Library Path for the same WebSphere MQ Messaging Providers.

Given that I'm running a 64-bit JVM on a 64-bit OS with a 64-bit MQ, I set the NLP to: -

C:\Program Files\IBM\WebSphere MQ\java\lib64

using some Jython: -

AdminTask.manageWMQ('"WebSphere MQ Resource Adapter(cells/PSCell1/nodes/Node1|resources.xml#J2CResourceAdapter_1434712637566)"', '[-nativePath "C:/Program Files/IBM/WebSphere MQ/java/lib64 " ]') 

Of course, I did also need to remember that it's necessary to restart WAS not once but TWICE, in order to avoid: -

[20/06/15 06:08:31:455 BST] 00000092 ActivationSpe E   J2CA0164E: The lookup of the Destination with JNDI Name CustomerService/MQJmsImport1_MQ_RECEIVE_D failed. The Destination is required by the Activation Specification. The lookup failed due to exception Exception occurred while the JNDI NamingManager was processing a javax.naming.Reference object. [Root exception is javax.naming.NamingException: WMSG2003E: It was not possible to lookup the specified WebSphere MQ administered object because the application server needs a restart to register an update to the WebSphere MQ installation.]
Caused by: javax.naming.NamingException: WMSG2003E: It was not possible to lookup the specified WebSphere MQ administered object because the application server needs a restart to register an update to the WebSphere MQ installation.

Once I restarted TWICE, all was back in order, which is nice :-)

Friday, 19 June 2015

IBM Integration Bus - Developing my interest ( and an SOA / ESB solution )

I have blogged about IBM Integration Bus (IIB) a fair bit recently, as I develop my expertise.

Most recently, I wrote: -

specifically as I'm busy developing an integration demonstration for my client, comprising IBM Business Process Manager, IBM Integration Bus, IBM WebSphere MQ and IBM DB2.

Today's post describes my experience creating a really really simple flow in IIB.

In the first instance, this flow represents a service that exposes a function from my mythical System of Record, actually a DB2 database table called EMPLOYEE.

My objective is to have this service running inside my Enterprise Service Bus (ESB), which is being delivered by IIB, and be callable in an asynchronous AND synchronous manner.

Initially, my service ( flow ) is only exposed via WebSphere MQ.

In principle, I have an Integration Node, called IB9NODE, which hosts an Integration Server, called IIB9, and a Queue Manager, called IB9QMGR.

The Queue Manager hosts a pair of Queues, CUSTOMER.INPUT and CUSTOMER.OUTPUT.

I will script the definition of the IIB components and the Queue Manager at a later date, to start with, I created them using the IIB Toolkit ( I'm using IIB 9 which has an automatic dependency on an underlying Queue Manager ).

Having set up IIB and the Queue Manager, I created my Queues: -



runmqsc IB9QMGR < defineQueues.msc 

5724-H72 (C) Copyright IBM Corp. 1994, 2014.
Starting MQSC for queue manager IB9QMGR.

AMQ8006: WebSphere MQ queue created.
AMQ8006: WebSphere MQ queue created.
2 MQSC commands read.
No commands have a syntax error.
All valid MQSC commands were processed.

My flow effectively joins the two Queues together, by way of a Compute Node. This Compute Node uses a Database Service, which does the clever stuff i.e. connecting to DB2 via ODBC and exposing one or more SQL operations - I'm "merely" doing a SELECT.

So, logically the client, be that MQ, the IIB Toolkit or, in the future, a BPEL flow hosted by IBM BPM, puts a message onto the CUSTOMER.INPUT Queue, the flow does the heavy lifting to retrieve the selected row from DB2, and places the output onto the CUSTOMER.OUTPUT Queue.

This is what my flow looks like: -

My Input Node is configured to use the CUSTOMER.INPUT Queue: -

which is configured parse a message in the JSON format: -

Here's an example message in JSON format: -


My Compute Node is configured to bind to the SAMPLE ODBC datasource ( see my previous post re ODBC fun and games ): -

The Compute Node contains some basic generated ESQL, which I've subtly modified: -


CREATE COMPUTE MODULE customerService_Compute

DECLARE dbResultSet ROW;
DECLARE dbResultSetRef REFERENCE TO dbResultSet;

DECLARE rowRef REFERENCE TO dbResultSetRef.row;

SET empno = InputRoot.JSON.Data.EmployeeID;
CALL retrieveEmployee(empno, dbResultSetRef);

SET OutputRoot.XMLNSC.EMPLOYEE = rowRef;

SET J = CARDINALITY(InputRoot.*[]);
SET OutputRoot.*[I] = InputRoot.*[I];
SET I = I + 1;

SET OutputRoot = InputRoot;

I've highlighted the code that I added to the generated ESQL.

Of that, this is what the IIB Toolkit gave me when I created the Database Service against the SAMPLE database: -

DECLARE dbResultSet ROW;
DECLARE dbResultSetRef REFERENCE TO dbResultSet;

DECLARE rowRef REFERENCE TO dbResultSetRef.row;

CALL retrieveEmployee(empno, dbResultSetRef);

and this is what I added: -

SET empno = InputRoot.JSON.Data.EmployeeID;
SET OutputRoot.XMLNSC.EMPLOYEE = rowRef;

The first line: -


sets up a variable called, imaginatively, empno as type CHARACTER.

The second line: -

SET empno = InputRoot.JSON.Data.EmployeeID;

assigns the value of the JSON object EmployeeID to the empno variable - the InputRoot "variable" relates to the entire incoming MQ message, and uses the JSON parser to retrieve the EmployeeID object.

The third line: -

SET OutputRoot.XMLNSC.EMPLOYEE = rowRef;

does almost the reverse - it assigns the value of rowRef ( as retrieved by the Database Service and stored as a row in a Result Set ) to the OutputRoot "variable", using the XMLNSC parser to create the EMPLOYEE XML message.

It's this EMPLOYEE message that is put onto the outgoing CUSTOMER.OUTPUT Queue.

Finally, here's me testing the flow, using the MQ samples amqsput and amqsget : -


/opt/mqm/samp/bin/amqsput CUSTOMER.INPUT IB9QMGR

Sample AMQSPUT0 start
target queue is CUSTOMER.INPUT


/opt/mqm/samp/bin/amqsget CUSTOMER.OUTPUT IB9QMGR

Sample AMQSGET0 start

I then went a little further, and changed my Compute Node to return a JSON object: -

SET OutputRoot.JSON.Data.Employee = rowRef;

Having redeployed the flow to the Integration Server ( on the Integration Node ) from the Toolkit, I re-tested it using the MQ samples: -

/opt/mqm/samp/bin/amqsput CUSTOMER.INPUT IB9QMGR

Sample AMQSPUT0 start
target queue is CUSTOMER.INPUT

/opt/mqm/samp/bin/amqsget CUSTOMER.OUTPUT IB9QMGR

Sample AMQSGET0 start
message <{"Employee":{"row":{"EMPNO":"000100","FIRSTNME":"THEODORE","LASTNAME":"SPENSER"}}}>
message <{"Employee":{"row":{"EMPNO":"000200","FIRSTNME":"DAVID","LASTNAME":"BROWN"}}}>

So, that's it for now ....

Next I need to create my "client" which will be an SCA module hosted on IBM BPM Advanced ( aka Process Server ) using a BPEL flow, which will use JMS to post the input message onto the CUSTOMER.INPUT Queue, and monitor the CUSTOMER.OUTPUT Queue for the resulting message.

I'll create the SCA module using IBM Integration Designer, and leverage the built-in Process Server integrated test environment.

When time allows, I'll then create a BPMN Process Application ( using Process Designer ) which will again be hosted on a BPM Advanced Process Server. This BPMN application will provide the user interface, where the end-user ( perhaps a service agent in a call centre or perhaps an employee via a self-service mobile application ) would enter the required employee ID and get back the resulting record.

From little acorns do large oak trees grow ......

For reference, we have this: -

and, purely for the record, there's a nice video here: -