Every day is a school day - WebSphere Application Server to WebSphere MQ via TLS 1.2

Following my previous posts: -

Using OpenSSL to connect via a specific SSL/TLS cipher

WAS Scripting and Syntax Errors - Tear your hair out

CWPKI0672E: Alias "" is not a personal certificate in key store "CellDefaultKeyStore".

I had one outstanding question - why did WebSphere MQ throw up nasty certificate validation errors when I attempted to start a Message Driven Bean that uses a JMS Activation Specification to connect, via TLS 1.2, to an encrypted Channel.

The WAS logs were chock full of errors: -

Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9204: Connection to host 'bpm856.uk.ibm.com(1420)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[java.net.SocketException: Broken pipe],3=bpm856.uk.ibm.com/192.168.33.100:1420 (bpm856.uk.ibm.com),4=SSLSocket.startHandshake,5=default]],3=bpm856.uk.ibm.com(1420),5=RemoteTCPConnection.protocolConnect]

Caused by: com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[java.net.SocketException: Broken pipe],3=bpm856.uk.ibm.com/192.168.33.100:1420 (bpm856.uk.ibm.com),4=SSLSocket.startHandshake,5=default]

Caused by: javax.net.ssl.SSLHandshakeException: java.net.SocketException: Broken pipe

[10/09/15 16:14:08:647 BST] 00000088 SibMessage    W   [:] CWSJY0003W: MQJCA4023: Startup reconnection failed for ActivationSpec 'javax.jms.Queue:jms/TESTQ@TESTQM <-1279839562>'. Exception details: '
                       Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ0018: Failed to connect to queue manager 'TESTQM' with connection mode 'Client' and host name 'bpm856.uk.ibm.com(1420)'.
Check the queue manager is started and if running in client mode, check there is a listener running. Please see the linked exception for more information.

     Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').

when I attempted to start the MDB.

This was what I saw in the MQ Queue Manager error log: -

cat ~/qmgrs/TESTQM/errors/AMQERR01.LOG 

AMQ9633: Bad SSL certificate for channel '????'.

EXPLANATION:
A certificate encountered during SSL handshaking is regarded as bad for one of
the following reasons: 
(a) it was formatted incorrectly and could not be validated 
(b) it was formatted correctly but failed validation against the Certification
  Authority (CA) root and other certificates held on the local system 
(c) it was found in a Certification Revocation List (CRL) on an LDAP server 
(d) a CRL was specified but the CRL could not be found on the LDAP server 
(e) an OCSP responder has indicated that it is revoked 

The channel is '????'; in some cases its name cannot be determined and so is
shown as '????'. The remote host is 'bpm856 (192.168.33.100)'. The channel did
not start. 

The details of the certificate which could not be validated are
'[Class=]GSKVALMethod::X509[Issuer=]CN=bpm856.uk.ibm.com,OU=Root
Certificate,OU=BAMCell1,OU=Dmgr,O=IBM,C=US[#=]0dfe1775e064[Subject=]CN=bpm856.uk.ibm.com,OU=BAMCell1,OU=Dmgr,O=IBM,C=US[Class=]GSKVALMethod::PKIX[Issuer=]CN=bpm856.uk.ibm.com,OU=Root
Certific'. 

The certificate validation error was 575010.

Through trial and error, and the essential intervention of a friend, we realised that the problem was relatively simple :-)

We're using a SSL Configuration and a Dynamic Outbound Endpoint SSL Configuration to ensure that WAS connects to MQ on a specific Channel with the MQ Signer Certificate.

In essence, WAS is, by default, sending its personal certificate, signed by WAS' own signer certificate to MQ.

Now the MQ Queue Manager Channel has a setting - SSLCAUTH - which is set to OPTIONAL.

If this was set to REQUIRED, then I'd expect MQ to require WAS to provide a certificate, for two-way client authentication.

However, I assumed that OPTIONAL was ... optional.

Therefore, because WAS is sending a default certificate, MQ is expecting to validate it - and is unable to do so.

There are two solutions: -

Weak Security

Change the WAS configuration to AVOID sending the default certificate: -

specifically this pertains to the Default Client Certificate Alias used within the SSL Configuration. In the screenshot above, the alias needs to be set to (none) rather than (default).

Stronger Security

Export the Signer Certificate from WAS e.g.

AdminTask.extractSignerCertificate('[-keyStoreName CellDefaultTrustStore -keyStoreScope (cell):BAMCell1 -certificateFilePath /tmp/signer.cer -base64Encoded true -certificateAlias root ]')

and import it into the MQ key store: -

runmqckm -cert -add -db ~/SSL/keystore.kdb -pw passw0rd -file /tmp/signer.cer -format ascii

giving us this: -

runmqckm -cert -list -db ~/SSL/keystore.kdb -pw passw0rd 

Certificates in database /var/mqm/SSL/keystore.kdb:
   "CN=bpm856.uk.ibm.com, OU=Root Certificate, OU=BAMCell1, OU=Dmgr, O=IBM, C=US"
   ibmwebspheremqtestqm

and then refresh the MQ SSL security configuration: -

echo "REFRESH SECURITY TYPE(SSL)" | runmqsc $QMGR

( Note that this command will terminate any active SSL/TLS channels, so take care to run it at a suitable time )

Conclusion

Once either approach is followed, the MDB starts happily, and works a treat.

For the record, this was immensely useful: -

Diagnostic improvements to MQ SSL/TLS error AMQ9633

as was this: -

Using SSL to Connect to a WebSphere Application Server with a WebSphere MQ Queue Manager