IBM BPM Advanced 8.5.5 - CWWBF0057E WSVR0102E and WSVR0195E seen when trying to stop a SCA Module

We saw this today: -

[15/09/15 16:28:39:474 BST] 00000159 AdminHelper   A   ADMN1010I: An attempt is made to stop the IRterieveProductEligitbilityEmulatorApp application. (User ID = defaultWIMFileBasedRealm/c03999)

[15/09/15 16:28:39:475 BST] 00000159 CompositionUn A   WSVR0192I: Stopping composition unit WebSphere:cuname=IRterieveProductEligitbilityEmulatorApp in BLA WebSphere:blaname=IRterieveProductEligitbilityEmulatorApp.

[15/09/15 16:28:39:478 BST] 00000159 ProcessContai E   CWWBF0057E: The application cannot be stopped because of existing process instances.: [com.ibm.bpe.database.ProcessTemplateB@35c52e15]

[15/09/15 16:28:39:481 BST] 00000159 ApplicationMg E   WSVR0102E: An error occurred stopping, IRterieveProductEligitbilityEmulatorApp

{1}

[15/09/15 16:28:39:481 BST] 00000159 CompositionUn E   WSVR0195E: Composition unit WebSphere:cuname=IRterieveProductEligitbilityEmulatorApp in BLA WebSphere:blaname=IRterieveProductEligitbilityEmulatorApp failed to stop.

This occurred when one of my team was attempting to stop/uninstall an Enterprise Archive (EAR) file, said EAR file contained an Service Component Architecture (SCA)  module, said SCA module contained a Business Process Execution Language (BPEL) component.

The message is pretty clear: -

[15/09/15 16:28:39:478 BST] 00000159 ProcessContai E   CWWBF0057E: The application cannot be stopped because of existing process instances.: [com.ibm.bpe.database.ProcessTemplateB@35c52e15]

The trouble is .... what do we do next ?

Well, this IBM Knowledge Centre article helps: -

Deleting process instances using Business Process Choreographer Explorer

In essence, one needs to use the Business Process Choreographer Explorer (BPC Explorer) : -

https://ibmbpm.uk.ibm.com:8443/bpc/faces/pages/Login.jsp

to locate, terminate and, perhaps, delete any existing process instances belonging to that particular BPEL component, before one can stop the SCA Module ( EAR file ) and delete / reinstall it.

So we logged into BPC Explorer as a user with WAS administration privileges, navigated to Process Instances > Administered by me and saw .... nothing, zip, nil, nada, niet, nien, zero :-(

The same was true when we logged in as wasadmin ....

Which was nice ......

Ah, yes, and the solution ?

Well, whilst wasadmin is a "super-user" ( by this, I do, of course, mean a user with the full WAS administration privileges ) in the world of WAS, in the world of BPM, there is one user with greater powers :-)

Yes, it's the Deployment Environment administrator ( deAdmin or similar ) is the user to which I refer ....

Once we logged in to BPC Explorer as deAdmin, we could see the two process instances, terminate them and then ( our choice ) delete them.

Once we did that, we were able to stop and uninstall the EAR / SCA module, and all was good.

It does, of course, raise the point .... in the world of IBM BPM, one cannot just assume that one can, as a WAS administrator, stop/uninstall an EAR file ....

There's always the potential of existing, running process instances, which need to be managed, BY THE BUSINESS, in terms of error handling, completion, migration, termination, deletion etc. and BPC Explorer is the place to go ....

Good to know :-)

"Network is unreachable" with Red Hat Enterprise Linux

I had a PEBCAK moment earlier this evening.

I was trying to work out why I was getting: -

Network is unreachable

when I was trying to contact ( ping ) a host external to my Red Hat Enterprise Linux VM.

The VM is running under VMware Fusion on my Mac, and I'm using Network Address Translation (NAT).

I have a static IP address that falls into the subnet of my Mac's external WiFi network, configured as follows: -

cat /etc/sysconfig/network-scripts/ifcfg-eth0 

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.33.100
NETMASK=255.255.255.0
BROADCAST=192.168.33.255

I'd already checked the DNS configuration for my VM: -

cat /etc/resolv.conf 

; generated by /sbin/dhclient-script
search localdomain uk.ibm.com
nameserver 192.168.33.2

However, when I switched back to using DHCP: -

cat /etc/sysconfig/network-scripts/ifcfg-eth0 

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=dhcp

I got an IP address in the same subnet.

So it's NOT the address, and it's not the DNS configuration.

So what can it be, I hear you cry ?

Yep, you've guessed it :-)

I'd FORGOTTEN to set the default gateway :-)

For a static IP address, this needs to be set in one of two places.

For me, I did it thusly: -

cat /etc/sysconfig/network-scripts/ifcfg-eth0 

DEVICE=eth0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=static
IPADDR=192.168.33.100
NETMASK=255.255.255.0
BROADCAST=192.168.33.255
GATEWAY=192.168.33.2

which sets it for that particular interface - eth0.

I could've set it globally as follows: -

cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=bpm856.uk.ibm.com

GATEWAY=192.168.33.2

Thanks to Google for pointing me in the right direction : -

Unable to access the internet in centOS from a static IP

Every day is a school day - WebSphere Application Server to WebSphere MQ via TLS 1.2

Following my previous posts: -

Using OpenSSL to connect via a specific SSL/TLS cipher

WAS Scripting and Syntax Errors - Tear your hair out

CWPKI0672E: Alias "" is not a personal certificate in key store "CellDefaultKeyStore".

I had one outstanding question - why did WebSphere MQ throw up nasty certificate validation errors when I attempted to start a Message Driven Bean that uses a JMS Activation Specification to connect, via TLS 1.2, to an encrypted Channel.

The WAS logs were chock full of errors: -

Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9204: Connection to host 'bpm856.uk.ibm.com(1420)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[java.net.SocketException: Broken pipe],3=bpm856.uk.ibm.com/192.168.33.100:1420 (bpm856.uk.ibm.com),4=SSLSocket.startHandshake,5=default]],3=bpm856.uk.ibm.com(1420),5=RemoteTCPConnection.protocolConnect]

Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[java.net.SocketException: Broken pipe],3=bpm856.uk.ibm.com/192.168.33.100:1420 (bpm856.uk.ibm.com),4=SSLSocket.startHandshake,5=default]

Caused by: javax.net.ssl.SSLHandshakeException: java.net.SocketException: Broken pipe

[10/09/15 16:14:08:647 BST] 00000088 SibMessage    W   [:] CWSJY0003W: MQJCA4023: Startup reconnection failed for ActivationSpec 'javax.jms.Queue:jms/TESTQ@TESTQM <-1279839562>'. Exception details: '
                       Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'TESTQM' with connection mode 'Client' and host name 'bpm856.uk.ibm.com(1420)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.

     Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').

when I attempted to start the MDB.

This was what I saw in the MQ Queue Manager error log: -

cat ~/qmgrs/TESTQM/errors/AMQERR01.LOG 

AMQ9633: Bad SSL certificate for channel '????'.

EXPLANATION:
A certificate encountered during SSL handshaking is regarded as bad for one of
the following reasons: 
(a) it was formatted incorrectly and could not be validated 
(b) it was formatted correctly but failed validation against the Certification
  Authority (CA) root and other certificates held on the local system 
(c) it was found in a Certification Revocation List (CRL) on an LDAP server 
(d) a CRL was specified but the CRL could not be found on the LDAP server 
(e) an OCSP responder has indicated that it is revoked 

The channel is '????'; in some cases its name cannot be determined and so is
shown as '????'. The remote host is 'bpm856 (192.168.33.100)'. The channel did
not start. 

The details of the certificate which could not be validated are
'[Class=]GSKVALMethod::X509[Issuer=]CN=bpm856.uk.ibm.com,OU=Root
Certificate,OU=BAMCell1,OU=Dmgr,O=IBM,C=US[#=]0dfe1775e064[Subject=]CN=bpm856.uk.ibm.com,OU=BAMCell1,OU=Dmgr,O=IBM,C=US[Class=]GSKVALMethod::PKIX[Issuer=]CN=bpm856.uk.ibm.com,OU=Root
Certific'. 

The certificate validation error was 575010.

Through trial and error, and the essential intervention of a friend, we realised that the problem was relatively simple :-)

We're using a SSL Configuration and a Dynamic Outbound Endpoint SSL Configuration to ensure that WAS connects to MQ on a specific Channel with the MQ Signer Certificate.

In essence, WAS is, by default, sending its personal certificate, signed by WAS' own signer certificate to MQ.

Now the MQ Queue Manager Channel has a setting - SSLCAUTH - which is set to OPTIONAL.

If this was set to REQUIRED, then I'd expect MQ to require WAS to provide a certificate, for two-way client authentication.

However, I assumed that OPTIONAL was ... optional.

Therefore, because WAS is sending a default certificate, MQ is expecting to validate it - and is unable to do so.

There are two solutions: -

Weak Security

Change the WAS configuration to AVOID sending the default certificate: -

specifically this pertains to the Default Client Certificate Alias used within the SSL Configuration. In the screenshot above, the alias needs to be set to (none) rather than (default).

Stronger Security

Export the Signer Certificate from WAS e.g.

AdminTask.extractSignerCertificate('[-keyStoreName CellDefaultTrustStore -keyStoreScope (cell):BAMCell1 -certificateFilePath /tmp/signer.cer -base64Encoded true -certificateAlias root ]')

and import it into the MQ key store: -

runmqckm -cert -add -db ~/SSL/keystore.kdb -pw passw0rd -file /tmp/signer.cer -format ascii

giving us this: -

runmqckm -cert -list -db ~/SSL/keystore.kdb -pw passw0rd 

Certificates in database /var/mqm/SSL/keystore.kdb:
   "CN=bpm856.uk.ibm.com, OU=Root Certificate, OU=BAMCell1, OU=Dmgr, O=IBM, C=US"
   ibmwebspheremqtestqm

and then refresh the MQ SSL security configuration: -

echo "REFRESH SECURITY TYPE(SSL)" | runmqsc $QMGR

( Note that this command will terminate any active SSL/TLS channels, so take care to run it at a suitable time )

Conclusion

Once either approach is followed, the MDB starts happily, and works a treat.

For the record, this was immensely useful: -

Diagnostic improvements to MQ SSL/TLS error AMQ9633

as was this: -

Using SSL to Connect to a WebSphere Application Server with a WebSphere MQ Queue Manager

CWPKI0672E: Alias "" is not a personal certificate in key store "CellDefaultKeyStore".

If you see: -

CWPKI0672E: Alias "" is not a personal certificate in key store "CellDefaultKeyStore".

when running a Jython script such as: -

cellID=AdminControl.getCell()
configAlias="WAS_to_WMQ"
cipher="SSL_RSA_WITH_AES_128_CBC_SHA256"

AdminTask.createSSLConfig('[-alias '+configAlias+' -type JSSE -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+' -serverKeyAlias -clientKeyAlias -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers '+cipher+' ]')

which returned: -

WASX7015E: Exception running command: "AdminTask.createSSLConfig('[-alias '+configAlias+' -type JSSE -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+' -serverKeyAlias -clientKeyAlias -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers '+cipher+' ]')"; exception information:
com.ibm.websphere.management.cmdframework.CommandValidationException: CWPKI0672E: Alias "" is not a personal certificate in key store "CellDefaultKeyStore".

consider stopping/restarting the wsadmin shell in order to create a nice clean new shell.

**UPDATE**

Or simply avoid trying to specify null aliases - instead of the above, use this: -

AdminTask.createSSLConfig('[-alias '+configAlias+' -type JSSE -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+'  -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers '+cipher+' ]')

thus avoiding the use of -serverKeyAlias and -clientKeyAlias altogether ( the objective is to ensure that both are NULL in order to avoid WAS > MQ connectivity problems over TLS ).

**UPDATE**

This old, but good, IBM document: -

PK86031: ADMINTASK.MODIFYSSLCONFIG UNABLE TO MODIFY NEWLY ADDED CERT DATA IN WORKSPACE TEMP, ERROR CWPKI0672E

led me to this solution, which is nice :-)

The text makes it more clear: -

...

Certificates added using AdminTask can not be modified with AdminTask.modifySSLConfig unless an AdminConfig.save() is performed before the modifySSLConfig.

Example error : CWPKI0672E: Alias "test_alias" is not a personal certificate in key store "CellDefaultKeyStore".

The modifySSLConfig command should read the key.p12 file from the workspace temp  - if there is any - and not just from the config repository.

The problem with executing the save operation is as follows:

For stable automation it is important that one script is executed as a unit.

If one command fails in this script, the whole commands in this script could be "rolled back". This is perfectly possible with the design of wsadmin. You execute your wsadmin commands and they do the changes only in your workspace temp. If one of your commands fail and you exit wsadmin, none of your previously executed commands will get saved.  Only at the end of your procedure you call the save operation and all changes get saved together.

However, if I have to do a save operation between these commands the first changes are already commited to the config repository although it is not assured that the next commands will succeed.

This could end in an inconsistent configuration state.

...

Whilst it's not 100% the same as my situation, I had had the wsadmin "shell" open for some time, and had also been deleting stuff from within the Integrated Solutions Console, meaning that the temporary configuration repository within the shel was likely to be out-of-sync with the cell configuration.

Bottom line, restarting the wsadmin client did the trick :-)

WAS Scripting and Syntax Errors - Tear your hair out

I've just posted a hair-tearing post to the Global WebSphere Community here: -

WAS Scripting and Syntax Errors - Tear your hair out

detailing my fun with a Jython script, a syntax error and WASX7122E.

As ever, it was a PEBCAK :-)

Enjoy :-)

Using OpenSSL to connect via a specific SSL/TLS cipher

We're busy setting up TLS 1.2 encryption for WebSphere MQ 8, forcing all connections ( from WebSphere Application Server, IBM Integration Bus etc. ) to be encrypted, via a dedicated SVRCONN Channel.

The MQ setup script includes the following: -

...

QMGR=TESTQM
QMGRPORT=1420
MQCIPHERSPEC=TLS_RSA_WITH_AES_128_CBC_SHA256

echo "DEFINE CHANNEL(TEST.QMGR.SVRCONN) CHLTYPE(SVRCONN) SSLCIPH("$MQCIPHERSPEC") REPLACE" | runmqsc $QMGR
echo "ALTER CHANNEL(TEST.QMGR.SVRCONN) CHLTYPE(SVRCONN) SSLCAUTH(OPTIONAL)" | runmqsc $QMGR
echo "DIS CHANNEL(TEST.QMGR.SVRCONN) CHLTYPE(SVRCONN)" | runmqsc $QMGR
...

meaning that connectivity to this specific Channel will use the TLS_RSA_WITH_AES_128_CBC_SHA256 cipher.

Having completed the configuration, I wanted to validate the connectivity, using OpenSSL, which is built into my server's OS ( Red Hat Enterprise Linux ).

openssl s_client -tls1_2 -connect `hostname`:1420

CONNECTED(00000003)
depth=0 CN = bpm856.uk.ibm.com
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = bpm856.uk.ibm.com
verify return:1
---
Certificate chain
 0 s:/CN=bpm856.uk.ibm.com
   i:/CN=bpm856.uk.ibm.com
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIB2zCCAUSgAwIBAgIEVfA5qjANBgkqhkiG9w0BAQUFADAcMRowGAYDVQQDExFi
cG04NTYudWsuaWJtLmNvbTAeFw0xNTA5MDkxMzUyNDJaFw0zNTA5MDQxMzUyNDJa
MBwxGjAYBgNVBAMTEWJwbTg1Ni51ay5pYm0uY29tMIGfMA0GCSqGSIb3DQEBAQUA
A4GNADCBiQKBgQCjEQKO/DA4y1gIv0wosNiAQx/V5vqaQHW8pEgy+mzJ/wcCdDRm
lqtm/E4JtAlbway70+OPdunKRMF54zizttXSWiDxrjFM8MRXJKGEsQfo2RpsKpUf
1YC0TRE8FUiUUCLIQmpLJ1aeW5RoxOxTQQwn97Rp0ULj7nq95anxgFuzDQIDAQAB
oyowKDATBgNVHSMEDDAKgAgPevSoHR450jARBgNVHQ4ECgQID3r0qB0eOdIwDQYJ
KoZIhvcNAQEFBQADgYEALGop6W2G8W/vlyf/0zBzZ0SupScUVQ875CIqE8Dm3wW6
pBn7HTDBGLIESJ4oQ0CdseiJEStBxmBtW4klotgL0A9C3nh2WNnSOxs1W1UqgCBB
fPQdWHU7iwogplgdfyb0xgWbwiobRoej8aOxMgL2yOyoojxFgspHrmMySEAjQvw=
-----END CERTIFICATE-----
subject=/CN=bpm856.uk.ibm.com
issuer=/CN=bpm856.uk.ibm.com
---
Acceptable client certificate CA names
/CN=bpm856.uk.ibm.com
---
SSL handshake has read 742 bytes and written 491 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA256
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : AES128-SHA256
    Session-ID: 835400002792AB6982E40F6F59B50F396703953B58585858D875F0550000000A
    Session-ID-ctx: 
    Master-Key: D20745FE1E201620E7EC9B209D2858059E5CC7D2A68AE7D8B40CACAFED2767B891A9330156EAB5F9E46E151D3BCA3B27
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1441822168
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---

To further ensure that I could only connect with one cipher, I narrowed down my OpenSSL command: -

openssl s_client -tls1_2 -connect `hostname`:1420 -cipher 'TLS_RSA_WITH_AES_128_CBC_SHA256'

which returned: -

error setting cipher list
140245232068424:error:1410D0B9:SSL routines:SSL_CTX_set_cipher_list:no cipher match:ssl_lib.c:1314:

( Note that I'd specified the actual cipher - TLS_RSA_WITH_AES_128_CBC_SHA256 - in the command )

I then checked the online man page for the -ciphers option: -

https://www.openssl.org/docs/man1.0.1/apps/ciphers.html

*UPDATED 24/10/2020*

https://www.openssl.org/docs/manmaster/man1/ciphers.html

*UPDATED 24/10/2020*

which did the trick: -

openssl s_client -tls1_2 -connect `hostname`:1420 -cipher 'AES128-SHA256'

returns: -

...
CONNECTED(00000003)
...

SSL handshake has read 736 bytes and written 343 bytes

...

New, TLSv1/SSLv3, Cipher is AES128-SHA256

...

    Protocol  : TLSv1.2
    Cipher    : AES128-SHA256

...

DB2 HADR - Updating Host Names

Following my earlier posts, I've now gone to another level, cloning my DB2 HADR server ( the original standby server ) and thus creating TWO DB2 servers, one dedicated as primary and one dedicated as standby.

Of course, the delights of HADR mean that I can switch back and forth, and that's the objective - I want to have a single VM hosting WebSphere Application Server ( IBM Business Monitor ) and TWO VMs hosting DB2, so that I can properly test failover by physically turning off a VM ( shutdown -h now ).

Having made all the necessary OS and DB2 / HADR changes, the only thing that I noticed was the the DB2 catalog was out-of-date.

This is what I had: -

db2 list db directory

 System Database Directory

 Number of entries in the directory = 2

Database 1 entry:

 Database alias                       = COGNOS
 Database name                        = COGNOS
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              = IBM Cognos Content Store
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = bpm856
 Alternate server port number         = 60006

Database 2 entry:

 Database alias                       = MONITOR
 Database name                        = MONITOR
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = bpm856
 Alternate server port number         = 60006

whereas I expected the Alternate server hostname to be either db2one or db2two, depending upon at which box I looked.

How do I fix this ??

I ask Google :-)

How to change the hostname in DB2 HADR and TSA environment system

which led me to this: -

Primary

db2 "update alternate server for database cognos using hostname db2one port 60006"
db2 "update alternate server for database monitor using hostname db2one port 60006"

Standby

db2 "update alternate server for database cognos using hostname db2two port 60006"
db2 "update alternate server for database monitor using hostname db2two port 60006"

Now the catalog looks rosy :-)

Primary

 System Database Directory

 Number of entries in the directory = 2

Database 1 entry:

 Database alias                       = COGNOS
 Database name                        = COGNOS
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              = IBM Cognos Content Store
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2two
 Alternate server port number         = 60006

Database 2 entry:

 Database alias                       = MONITOR
 Database name                        = MONITOR
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2two
 Alternate server port number         = 60006

Standby

 System Database Directory

 Number of entries in the directory = 2

Database 1 entry:

 Database alias                       = COGNOS
 Database name                        = COGNOS
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              = IBM Cognos Content Store
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2one
 Alternate server port number         = 60006

Database 2 entry:

 Database alias                       = MONITOR
 Database name                        = MONITOR
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2one
 Alternate server port number         = 60006

And DB2 HADR is all clean and green.

Now to update WAS to use the correct hostnames via JDBC data sources :-)