AMQ9660: SSL key repository: password stash file absent or unusable.

I've spent a few happy hours trying to resolve this one.

When attempting to read a message from a WebSphere MQ 8 Queue from WebSphere Application Server (WAS), using a Message Driven Bean (MDB), a JMS Activation Specification and a JMS Queue, I kept seeing this: -

-------------------------------------------------------------------------------
11/11/15 19:57:29 - Process(39916.4) User(mqm) Program(amqrmppa)
                    Host(nemdemo.uk.ibm.com) Installation(Installation1)
                    VRMF(8.0.0.0) QMgr(TESTQM)
                   
AMQ9660: SSL key repository: password stash file absent or unusable.

EXPLANATION:
The SSL key repository cannot be used because MQ cannot obtain a password to
access it. Reasons giving rise to this error include: 
(a) the key database file and password stash file are not present in the
  location configured for the key repository, 
(b) the key database file exists in the correct place but that no password
  stash file has been created for it, 
(c) the files are present in the correct place but the userid under which MQ is
  running does not have permission to read them, 
(d) one or both of the files are corrupt. 

The channel is '????'; in some cases its name cannot be determined and so is
shown as '????'. The channel did not start.
ACTION:
Ensure that the key repository variable is set to where the key database file
is. Ensure that a password stash file has been associated with the key database
file in the same directory, and that the userid under which MQ is running has
read access to both files. If both are already present and readable in the
correct place, delete and recreate them. Restart the channel.

I also noticed that WAS returned: -

     Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2397' ('MQRC_JSSE_ERROR').

     Caused by [2] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9204: Connection to host 'nemdemo.uk.ibm.com(1420)' rejected. [1=com.ibm.mq.jmqi.JmqiException[CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[Remote host closed connection during handshake],3=nemdemo.uk.ibm.com/192.168.1.113:1420 (nemdemo.uk.ibm.com),4=SSLSocket.startHandshake,5=default]],3=nemdemo.uk.ibm.com(1420),5=RemoteTCPConnection.protocolConnect]

     Caused by [3] --> Message : com.ibm.mq.jmqi.JmqiException: CC=2;RC=2397;AMQ9771: SSL handshake failed. [1=javax.net.ssl.SSLHandshakeException[Remote host closed connection during handshake],3=nemdemo.uk.ibm.com/192.168.1.113:1420 (nemdemo.uk.ibm.com),4=SSLSocket.startHandshake,5=default]

     Caused by [4] --> Message : javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

     Caused by [5] --> Message : java.io.EOFException: SSL peer shut down incorrectly

This all occurred as soon as I started the MDB.

The connectivity between WAS and MQ is encrypted using a CA-signed certificate, over a TLS 1.2 connection.

I had previously set up my Queue Manager to use TLS: -

echo "ALTER QMGR SSLKEYR('"/home/mqm/SSL/keystore.kdb"')" | runmqsc TESTQM
echo "DIS QMGR SSLKEYR" | runmqsc TESTQM
echo "REFRESH SECURITY TYPE(SSL)" | runmqsc TESTQM

and my Server Connection Channel to use a specific cipher: -

echo "DEFINE CHANNEL("TEST.QMGR.SVRCONN") CHLTYPE(SVRCONN) SSLCIPH("TLS_RSA_WITH_AES_128_CBC_SHA256") REPLACE" | runmqsc TESTQM
echo "ALTER CHANNEL("TEST.QMGR.SVRCONN") CHLTYPE(SVRCONN) SSLCAUTH(OPTIONAL)" | runmqsc TESTQM

Reading the message explanation above, I spent time looking at the key store / stashed password etc.

ls -al /home/mqm/SSL/

total 28
drwxrwxr-x  2 mqm mqm  4096 Nov 11 13:02 .
drwx------. 4 mqm mqm  4096 Nov 11 19:02 ..
-rwxrwxrwx  1 mqm mqm 10080 Nov 11 15:11 keystore.kdb
-rwxrwxrwx  1 mqm mqm    80 Nov 11 15:11 keystore.rdb
-rwxrwxrwx  1 mqm mqm   129 Nov 11 15:11 keystore.sth

checking that the MQ user mqm had full access to the key store.

I also checked that the stashed password worked OK: -

runmqakm -cert -list -db /home/mqm/SSL/keystore.kdb -stashed

Certificates found
* default, - personal, ! trusted, # secret key
! "CN=uk-WIN-AJ9S32NP29C-CA, DC=uk, DC=ibm, DC=com"
*- nemdemo.uk.ibm.com

so that ruled that one out.

I then read a few IBM PMRs and found a potential hint, suggesting that the extension of the keystore database, as defined in the Queue Manager, does NOT need to be specified.

I updated the Queue Manager configuration: -

echo "ALTER QMGR SSLKEYR('/home/mqm/SSL/keystore')" | runmqsc TESTQM

echo "REFRESH SECURITY TYPE(SSL)" | runmqsc TESTQM

and, guess what, it worked :-)