Saturday, 30 May 2015

WebSphere Application Server, TLS 1.2 and DB2

Some more blogging over at the WebSphere User Group ( aka Global WebSphere Community ), following my continuing voyage of discovery in the world of Transport Layer Security (TLS): -

I've been working through the configuration of Transport Layer Security (TLS) 1.2 between DB2 and WebSphere Application Server (WAS).

I've learned a heck of a lot about this in the past 48 hours, but the key aspect is that it's necessary to configure BOTH DB2 *AND* WAS to support TLS 1.2

Things that make you go "Hmmm" - #432 - WebSphere Application Server Transaction and Partner Logs

Over the past few weeks, I've written about my experiences configuring IBM Business Process Manager and IBM Business Monitor to connect via a TLS-encrypted tunnel to IBM DB2: -

and am just about to create a post covering the experiences learned whilst configuring WebSphere Application Server to support the current latest Transport Layer Security (TLS) 1.2.

However, I hit a small glitch....

Whilst validating my current setup ( IBM Business Monitor 8.5.6 on WAS ND connecting via TLS 1.0 to DB2 ), I noted the following exception in one of my cluster member logs ( specifically the AppTarget ): -

[30/05/15 06:54:23:906 BST] 00000065 RecoveryManag I   WTRN0135I: Transaction service recovering no transactions.
[30/05/15 06:54:23:917 BST] 00000065 RecoveryManag A   WTRN0134I: Recovering 1 XA resource manager(s) from the transaction partner logs
[30/05/15 06:54:23:954 BST] 00000065 XARecoveryDat A   WTRN0151I: Preparing to call xa recover on XAResource: Monitor_Database
[30/05/15 06:54:24:891 BST] 00000065 DMAdapter     I getAnalysisEngine FFDC1009I: Analysis Engine using data base: /opt/IBM/WebSphere/AppServer/properties/logbr/ffdc/adv/ffdcdb.xml
[30/05/15 06:54:24:897 BST] 00000065 FfdcProvider  W logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/logs/ffdc/AppClusterMember1_c432d1b_15.05.30_06.54.24.8794642297944511548081.txt 1298
[30/05/15 06:54:24:939 BST] 00000065 FfdcProvider  W logIncident FFDC1003I: FFDC Incident emitted on /opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/logs/ffdc/AppClusterMember1_c432d1b_15.05.30_06.54.24.9204315923791292855617.txt 310
[30/05/15 06:54:24:942 BST] 00000065 J2CXAResource W   J2CA0061W: Error creating XA Connection and Resource DSRA8100E: Unable to get a XAConnection from the DataSource jdbc/wbm/MonitorDatabase. with SQL State : 08001 SQL Code : -4499

Caused by: [jcc][t4][2043][11550][4.18.60] Exception Error opening socket to server on port 60,006 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499

java.sql.SQLNonTransientException: [jcc][t4][2043][11550][4.18.60] Exception Error opening socket to server on port 60,006 with message: Connection refused. ERRORCODE=-4499, SQLSTATE=08001 DSRA0010E: SQL State = 08001, Error Code = -4,499

Caused by: Connection refused

Having gone through the configuration with a fine tooth comb ( whatever one of those is ), I could NOT find ANY reference to port  60006 anywhere.

For the record, port 60006 is the non-TLS port that I'd previously used, before switching to port 60007 for a TLS-encrypted connection.

After much trial and quite a lot of error, I re-read the log, specifically these two lines: -

[30/05/15 06:54:23:917 BST] 00000065 RecoveryManag A   WTRN0134I: Recovering 1 XA resource manager(s) from the transaction partner logs
[30/05/15 06:54:23:954 BST] 00000065 XARecoveryDat A   WTRN0151I: Preparing to call xa recover on XAResource: Monitor_Database

which started me thinking about the Transaction Manager.

What, I wondered, was the possibility that the OLD pre-TLS configuration was still persisted in a transaction that had previously NOT completed before I switched the configuration across ?

I did some further digging ( using the command fgrep -R 60006 * ) inside the directory that hosts the Transaction, Recovery and Partner logs for the AppCluster: -

cd /opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/tranlog/BAMCell1/AppSrv01Node/AppClusterMember1/transaction

and found two binary files: -


here: -


both of which contained references to the string 60006.

That confirmed my suspicion.

Now there's a third file in this directory, sensibly named: -


That's there for a VERY good reason - one should NEVER delete the Transaction or Partner Log files.


Having said NEVER, this is MY own test environment with NO important or critical data - if I break things, I simply rebuild the WAS cell, which takes ~30 minutes.

So, ignoring my own ( and IBM's ) advice, I delete the Partner Logs: -

cd /opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/tranlog/BAMCell1/AppSrv01Node/AppClusterMember1/transaction/partner
rm -Rf *

having shut down the AppCluster.

Quelle surprise, when I restarted the cluster, there were no failed transactions to recover, and WAS came up clean and green with NO JDBC exceptions.


Thinking about it after the event, I probably could have achieved the same thing by re-opening port 60006 on DB2, which I'd previously disabled using the db2set command as follows: -

db2set DB2COMM=SSL

thus overriding the previous configuration: -


For the record, the SSL value means that DB2 observes the SSL service configuration within the Database Manager: -

SSL service name                         (SSL_SVCENAME) = db2c_ssl

whereas TCPIP means that it observes the TCPIP service configuration: -

TCP/IP Service name                          (SVCENAME) = db2c_db2inst1

In each case, the Service Name is inferred from /etc/services which ensures that the instance is listening on the appropriate ports: -

DB2_db2inst1 60000/tcp
DB2_db2inst1_1 60001/tcp
DB2_db2inst1_2 60002/tcp
DB2_db2inst1_3 60003/tcp
DB2_db2inst1_4 60004/tcp
DB2_db2inst1_END 60005/tcp
db2c_db2inst1 60006/tcp
db2c_ssl 60007/tcp

So, had I enabled BOTH services, WAS would've been able to connect via a non-TLS connection to port 60006 and the transaction would have been recovered / completed.

Life is, as ever, a learning curve :-)

For future reference, there's plenty of good material covering the WAS Java Transaction Service, including this: -

which does cover the costs and benefits of deleting the Transaction and Partner Logs, especially in the context of WebSphere Process Server ( now IBM BPM ).

So, again, do NOT NOT NOT delete Tran/Partner Logs unless you really really really know what you're doing.

Friday, 29 May 2015

IBM HTTP Server, Transport Layer Security and Google Chrome

In another of my occasional posts for the Global WebSphere Community (GWC), aka the WebSphere User Group, I've just submitted an article covering my experiences with IBM HTTP Server, SSL/TLS and Chrome.

It's on the GWC site here: -

I hope it's of some use :-)

Friday, 15 May 2015

Business Process Management Design Guide: Using IBM Business Process Manager

IBM® Business Process Manager (IBM BPM) is a comprehensive business process management (BPM) suite that provides visibility and management of your business processes. IBM BPM supports the whole BPM lifecycle approach: 

• Discover and document
• Plan
• Implement
• Deploy
• Manage
• Optimize
Process owners and business owners can use this solution to engage directly in the improvement of their business processes.

IBM BPM excels in integrating role-based process design, and provides a social BPM experience. It enables asset sharing and creating versions through its Process Center. The Process Center acts as a unified repository, making it possible to manage changes to the business processes with confidence.

IBM BPM supports a wide range of standards for process modeling and exchange. Built-in analytics and search capabilities help to further improve and optimize the business processes.

This IBM Redbooks® publication provides valuable information for project teams and business people that are involved in projects using IBM BPM. It describes the important design decisions that you face as a team. These decisions invariably have an effect on the success of your project.

These decisions range from the more business-centric decisions, such as which should be your first process, to the more technical decisions, such as solution analysis and architectural considerations.

Table of contents

Chapter 1. Introduction to successful business process management
Chapter 2. Approaches and process discovery
Chapter 3. Solution analysis and architecture considerations
Chapter 4. Security architecture considerations
Chapter 5. Design considerations and patterns
Chapter 6. Business-centric visibility
Chapter 7. Performance and IT-centric visibility

Thursday, 14 May 2015

Continuing to learn - IBM BPM and IBM Business Monitor to DB2 via SSL/TLS

I've written a couple of posts on the WebSphere User Group blog here: -

My next trick will be to force WebSphere Application Server (WAS) to use a specific encryption standard, namely TLS version 1.2.

In DB2, this can be enforced as follows: -

db2 update dbm config using SSL_VERSIONS TLSV12

for version 1.2 or: -

db2 update dbm config using SSL_VERSIONS TLSV1

for version 1.0, or: -

db2 update dbm config using SSL_VERSIONS NULL

to revert back to SSL.

If you set the parameter to null or TLSv1, the parameter enables support for TLS version 1.0 (RFC2246) and TLS version 1.1 (RFC4346).

Note: During SSL handshake, the client and the server negotiate and find the most secure version to use either TLS version 1.0 or TLS version 1.1. If there is no compatible version between the client and the server, the connection fails. If the client supports TLS version 1.0 and TLS version 1.1, but the server support TLS version 1.0 only, then TLS version 1.0 is used.
If you set the parameter to TLSv12 (RFC5246), the parameter enables support for TLS version 1.2. This setting is required to comply with NIST SP 800-131A.

If you set the parameter to TLSv12 and TLSv1, the parameter enables support for TLS version 1.2 with the option to fall back on TLS version 1.0 and 1.1.

All of the SSL-related settings can be queried thusly: -

db2 get dbm config | grep SSL

 SSL server keydb file                   (SSL_SVR_KEYDB) = /home/db2inst1/keystore.kdb
 SSL server stash file                   (SSL_SVR_STASH) = /home/db2inst1/keystore.sth
 SSL server certificate label            (SSL_SVR_LABEL) =
 SSL service name                         (SSL_SVCENAME) = db2c_ssl
 SSL cipher specs                      (SSL_CIPHERSPECS) = 
 SSL versions                             (SSL_VERSIONS) = 
 SSL client keydb file                  (SSL_CLNT_KEYDB) = 
 SSL client stash file                  (SSL_CLNT_STASH) = 

Note that we also have SSL_CIPHERSPECS to specify the cipher specifications that one wishes to use, as per this: -

and: -

Wednesday, 13 May 2015

IBM Cognos - Working with SSL/TLS Keystore

This is an ongoing voyage of discovery, as I seek to replicate my success: -

( this is a post that I authored for the WebSphere User Group on their Global WebSphere Community site )

with IBM Business Monitor.

Whilst the DB2 and WAS aspects ( configuring the DB2 instance and listener for SSL, updating the WAS JDBC data sources, adding the DB2 signer certificate into he WAS trust store etc. ) are the same, the Cognos BI engine is quite different.

I don't yet have it cracked, but I did discover a few more things about Cognos BI today, specifically in terms of where it keeps its own SSL/TLS key store.

It's here: -

-rw-r--r-- 1 wasadmin wasadmins 19728 May 13 16:07 /opt/IBM/WebSphere/AppServer/profiles/BAMCell1AppSrv01/cognos/SupClusterMember1/configuration/certs/CAMKeystore

Why do I know this ?

Because I wanted to test a hypothesis by adding the DB2 server's signer certificate to it.

This is how I first retrieved the signer certificate: -

openssl s_client -showcerts -connect localhost:60007 </dev/null > ~/db2.cer

and I happily verified the certificate: -

openssl x509 -fingerprint -noout -text -in ~/db2.cer

SHA1 Fingerprint=FC:BB:C1:24:4E:6E:B8:55:5B:33:87:69:C7:E2:10:E4:E6:0F:7A:CC
        Version: 3 (0x2)
        Serial Number: 1681898445175821098 (0x17574db98cfc932a)
    Signature Algorithm: sha1WithRSAEncryption
        Issuer: DC=com, DC=ibm, DC=uk,
            Not Before: May 11 10:01:51 2015 GMT
            Not After : May 11 10:01:51 2016 GMT
        Subject: DC=com, DC=ibm, DC=uk,
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (1024 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: critical
            X509v3 Subject Key Identifier: 
            X509v3 Authority Key Identifier: 

    Signature Algorithm: sha1WithRSAEncryption

However, when I tried to add it to the Cognos key store: -

/opt/IBM/WebSphere/AppServer/java/jre/bin/keytool -import -file ~/db2.cer -alias DB2 -keystore CAMKeystore -storepass MONITOR -storetype PKCS12

I saw this: -

keytool error: java.lang.Exception: Input not an X.509 certificate

Happily a quick Google search later, and I found this: -

which says, in part: -

While I agree with Ari's answer (and upvoted it :), I needed to do an extra step to get it to work with Java on Windows (where it needed to be deployed):

openssl s_client -showcerts -connect < /dev/null | openssl x509 -outform DER > derp.der

Before adding the openssl x509 -outform DER conversion, I was getting an error from keytool on Windows complaining about the certificate's format. Importing the .der file worked fine.

I re-retrieved the certificate from DB2: -

openssl s_client -showcerts -connect localhost:60007 </dev/null | openssl x509 -outform DER > ~/db2.cer

( adding in the relevant Hogwarts magic to get the resulting file in x509 DER ) and was then able to import it: -

/opt/IBM/WebSphere/AppServer/java/jre/bin/keytool -import -file ~/db2.cer -alias DB2 -keystore CAMKeystore -storepass MONITOR -storetype PKCS12

Owner:, DC=uk, DC=ibm, DC=com
Issuer:, DC=uk, DC=ibm, DC=com
Serial number: 17574db98cfc932a
Valid from: 11/05/15 11:01 until: 11/05/16 11:01
Certificate fingerprints:
 MD5:  81:B0:E7:81:A3:1B:79:64:07:1B:41:9E:7E:0A:F3:08
 SHA1: FC:BB:C1:24:4E:6E:B8:55:5B:33:87:69:C7:E2:10:E4:E6:0F:7A:CC
Trust this certificate? [no]:  y
Certificate was added to keystore

which is nice.

Did that fix my problem ? Alas, no, but it's another step on the journey to ......... ?

Tuesday, 12 May 2015

IBM Integration Designer - 101

I'm learning to get to grips with IBM Integration Designer (IID) at present, hence my post on the WUG here: -

Part of my self-enablement has come from a pair of tutorials, cannily called Hello World, that I found a few days ago: -

I was using older ( version 7.5.1 ) editions of these two tutorials, although they worked perfectly ( user errors notwithstanding !! ).

However, here's the more up-to-date source for the most recent versions of IBM BPM: -

Interestingly, Hello World seems to have disappeared with the latest version of the product.

In addition, BPM 8.5.6 also has some tutorials in the Knowledge Centre here: -

which is nice.

Monday, 11 May 2015

WebSphere Liberty Profile and IBM HTTP Server - Exchanging SSL Certificates

This one is for a friend of mine, TonyH, who asked this question earlier.

I don't claim to understand his requirements, but he asked: -

What's the best way to export certs from liberty so I can import them into the IHS keystore?

I'm running Liberty Profile on my Mac and IBM HTTP Server (IHS) on a Red Hat VM.

I started by downloading the Liberty Profile Runtime from here: -

which resulted in: -

-rw-r--r--@  1 davehay  staff  60261097 11 May 18:26 wlp-runtime-

and installed it: -

java -jar wlp-runtime-

to here: -


I then followed Oliver Rebmann's excellent blog post: -

to setup a SSL key store and certificate, as follows: -

cd /Users/davehay/Liberty/wlp/bin
./securityUtility createSSLCertificate --server=defaultServer --password=passw0rd --validity=365

and added the relevant configuration to my server's configuration: -

vi /Users/davehay/Liberty/wlp/usr/servers/defaultServer/server.xml

adding the lines highlighted below: -

<?xml version="1.0" encoding="UTF-8"?>
<server description="new server">

    <!-- Enable features -->

    <!-- To access this server from a remote client add a host attribute to the following element, e.g. host="*" -->
    <httpEndpoint id="defaultHttpEndpoint"
                  httpsPort="9443" />

    <keyStore id="defaultKeyStore" password="{xor}Lz4sLChvLTs=" />


and started Liberty: -

/Users/davehay/Liberty/wlp/bin/server start

Having validated that I could connect to Liberty on port 9443 ( see the httpsPort directive above ): -

I then used the openssl tool to retrieve the certificate from port 9443 to a file: -

openssl s_client -showcerts -connect localhost:9443 </dev/null > ~/liberty.cer

Having shipped the certificate file from the Mac to the Red Hat VM: -

scp ~/liberty.cer wasadmin@bpm856:~

I then imported it into the IHS key store: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -add -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd -file ~/liberty.cer -label liberty

and validated it thus: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -list -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd

Certificates found
* default, - personal, ! trusted, # secret key
! liberty

and: -

/opt/IBM/HTTPServer/bin/gskcapicmd -cert -details -db /opt/IBM/HTTPServer/ssl/keystore.kdb -pw passw0rd -label liberty

Label : liberty
Key Size : 2048
Version : X509 V3
Serial : 5550f616
Issuer : CN=,OU=defaultServer,O=ibm,C=us
Subject : CN=,OU=defaultServer,O=ibm,C=us
Not Before : May 11, 2015 7:33:58 PM GMT+01:00
Not After : May 10, 2016 7:33:58 PM GMT+01:00


The job, as they say, is a good 'un.

PS Thanks to Oliver for his insights ....

Saturday, 9 May 2015

Adding Ant to IBM Operational Decision Manager

In a departure from the norm, I'm NOT going to detail my recent experiences with Apache Ant and IBM ODM Ruls here, but instead direct you to my post on another site, the Global WebSphere Community (GWC), which hosts the WebSphere User Group.

So you want to know more about Ant and ODM ? Then please read my post here: -

and feel free to provide feedback here, there, or via Twitter.

Wednesday, 6 May 2015

Disaster recovery guidance for IBM Business Process Manager - An updated approach for IBM BPM V8.x

IBM® Business Process Manager (BPM) is a powerful tool for modeling and running an organization's most critical business processes. Over the past several years, features have been developed and verified to enable cross-site replication and recovery, achieving very sophisticated disaster-recovery objectives. Learn about the core principles that guide an infrastructure architect to design a successful disaster-recovery strategy.

Sunday, 3 May 2015

Reminder to Self - If you see SSL0279E again

Following on from an older post: -

if I ever see: -

SSL0279E: SSL Handshake Failed due to fatal alert from client. Client sent fatal alert [level 2 (fatal), description 46 (certificate_unknown)]

in the IHS error_log file again, do NOT waste time exporting certificates from WAS cell-default trust stores and importing them into the IHS KDB.

This is, again, in the context of connecting Process Center to Process Server, albeit with the most recent level of IBM BPM, 8.5.6.

The problem ABSOLUTELY comes from the fact that the Process Server AppCluster JVM is trying to connect to the Process Center AppCluster JVM ( from port 9447 to port 9443 ), as evidenced by this: -

[03/05/15 08:11:55:320 BST] 0000013c WSX509TrustMa E   CWPKI0022E: SSL HANDSHAKE FAILURE:  A signer with SubjectDN ", OU=PSCell1Node1, OU=Node1, O=IBM, C=US" was sent from target host:port "".  The signer may need to be added to local trust store "/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/PCCell1/trust.p12" located in SSL configuration alias "NodeDefaultSSLSettings" loaded from SSL configuration file "security.xml".  The extended error message from the SSL handshake exception is: "PKIX path building failed: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is: The certificate issued by, OU=Root Certificate, OU=PSCell1, OU=Dmgr, O=IBM, C=US is not trusted; internal cause is: Certificate chaining error".

in the Process Center's AppCluster JVM SystemOut.log.

The solution is to import the signer certificate from Process Center's IHS server into the cell-default trust store for the Process Server: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr02/bin/ -lang jython -user wasadmin -password passw0rd -host `hostname` -port 8883
AdminTask.retrieveSignerFromPort('[-keyStoreName CellDefaultTrustStore -keyStoreScope (cell):'+cellID+' -host -port 8443 -certificateAlias ProcessCenter -sslConfigName CellDefaultSSLSettings -sslConfigScopeName (cell):'+cellID+' ]')

Once done, and all is now good :-)

Friday, 1 May 2015

IBM Integration Designer - The filename, directory name, or volume label syntax is incorrect.

I saw this exception: -

[01/05/15 12:45:56:762 BST]     FFDC ProbeId:163 Reporter:java.lang.Class@7b57bfee
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(
                at java.lang.reflect.Method.invoke(
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(
                at java.lang.reflect.Method.invoke(
                at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(
                at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(
                at java.lang.reflect.Method.invoke(
                at org.eclipse.core.launcher.Main.invokeFramework(
                at org.eclipse.core.launcher.Main.basicRun(
Caused by: The filename, directory name, or volume label syntax is incorrect.

                ... 36 more

CapturedDataElements begin
CapturedDataElements end

 earlier whilst trying to create an IBM BPM Deployment Environment as a local Unit Test Environment for IBM Integration Designer 8.5.5 on Windows 7.

Under the covers, I'm using WebSphere Application Server, BPM Advanced 8.5.50 and Business Monitor

This was the command that I ran: -

"C:\IBM\WebSphere\AppServerbin\BPMConfig.bat" -create -de c:\

After much trial and much error, I realised the error of my ways.

In the properties file - - which I sourced from: -


( I'd previously unpacked the IID media into c:\temp\IID\install )

I had: -"C:\\IBM\WebSphere\\AppServer"

rather than: -\\IBM\WebSphere\\AppServer

In other words, the quotation symbols ( "" ) were getting in the way :-(

I'd rightly used the double-backslash ( \\ ) as required, BUT the quotations were NOT required.

Easy peasy.....