WebSphere Plugin and the Case of the GSK_ERROR_BAD_KEYFILE_PASSWORD

We've seen a few instances , where the WebSphere Plugin fails to communicate, via SSL, with WAS.

This manifests itself as Error 500 / HTTP500 when accessing WAS via IHS, using hostname OR service name.

Long story short, it looks like the Plugin SSL configuration files are getting "borked" by something.

The plugin log shows this: -

[22/Jun/2016:13:14:40.16292] 00d3008c 00000001 - ERROR: lib_security: logSSLError: str_security (gsk error 408):  GSK_ERROR_BAD_KEYFILE_PASSWORD
[22/Jun/2016:13:14:40.16294] 00d3008c 00000001 - ERROR: lib_security: initializeSecurity: Failed to initialize GSK environment. Secure transports are not possible.

One symptom is that you cannot query the keystore using the gskcapicmd command, as per this: -

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db plugin-key.kdb -stashed

which returns: -

CTGSK3026W The key file "plugin-key.kdb" does not exist or cannot be read.
CTGSK2016W An invalid database password was encountered.

The same error occurs if you use the correct keystore password e.g. WebAS.

We saw this problem even when we deleted the .KDB and .STH files ( see below ), and propagated them from the WAS cell via the Deployment Manager.

The problem appears to be related to the use of the gskcapicmd command to create certificate requests.

I *think* that, as some point, someone has created certificate requests for the Plugin, which has updated one or more of the related configuration files.

There are a number of files in play here: -

plugin-cfg.xml                                  Configuration file, generated by WAS, and propagated from the Deployment Manager to the IHS box
plugin-key.kdb                                  CMS key database, holding BOTH personal AND signer certificates ( keys and trusts )
plugin-key.sth                                   Encrypted stashed password file
plugin-key.crl                                    Certificate Recovation List
plugin-key.rdb                                  Certificate Request Database

As a test, we moved the .CRL and .RDB files into a different subdirectory, leaving just the .xml, .kdb and .sth files in place.

We were then able to query the .KDB without problems: -

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db plugin-key.kdb –stashed

Certificates found
* default, - personal, ! trusted, # secret key
!       "CN=was.uk.ibm.com, OU=RootCertificate, OU=test, OU=dmgr, O=ibm, O=co, C=uk"
-       default

/opt/ibm/HTTPServer/bin/gskcapicmd -cert -list -db plugin-key.kdb -pw WebAS

Certificates found
* default, - personal, ! trusted, # secret key
!       "CN=was.uk.ibm.com, OU=RootCertificate, OU=test, OU=dmgr, O=ibm, O=co, C=uk"
-       default

I did some more testing, moving the .CRL and .RDB files back and forth, and have concluded that it IS the .RDB file that "breaks" things.

Once we ended up with JUST the .XML, .KDB and .STH files in place ( in /opt/ibm/WebSphere/Plugins/config/ ), I was able to successfully navigate to WAS via IHS.

Bottom line, there's no need to use the IHS GSK commands ( gskcapicmd ) to request certificates for the WebSphere Plugin in the context of WAS.

If we need a personal WAS certificate, we can generate the Certificate Request using Jython scripts or the ISC, and WAS will take care of updating the KDB etc.

This is different to Plugin -> IBM Integration Bus, where there's no WAS to manage things for us.

The only time we'd ever need to use gskcapicmd against the Plugin KDB was if we wanted to mark the WAS personal certificate as default, in order to ensure that it was used for Plugin -> WAS connectivity, in the context of Mutual Authentication etc ( via the –setdefault  command ).