Oops, my WAS upgrade broke my TLS

I saw this earlier today: -

[30/03/16 11:35:53:371 BST] 00000001 ORBRas        E com.ibm.ws.orbimpl.transport.WSTransport createServerSocket P=152627:O=0:CT ORBX0390E: Cannot create listener thread. Exception=[ org.omg.CORBA.INTERNAL: CAUGHT_EXCEPTION_WHILE_CONFIGURING_SSL_SERVER_SOCKET, Exception=java.lang.IllegalArgumentException: Cannot support SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384 with currently installed providers vmcid: 0x49421000  minor code: 77  completed: No - received while attempting to open server socket on port 9403 ].

[30/03/16 11:35:53:397 BST] 00000001 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/ibm/WebSphereProfiles/ODMCell1Dmgr01/logs/ffdc/dmgr_498f4d6b_16.03.30_11.35.53.3748582431257869454898.txt com.ibm.ws.orbimpl.transport.WSTransport.startListening 805

[30/03/16 11:35:53:419 BST] 00000001 FfdcProvider  W com.ibm.ws.ffdc.impl.FfdcProvider logIncident FFDC1003I: FFDC Incident emitted on /opt/ibm/WebSphereProfiles/ODMCell1Dmgr01/logs/ffdc/dmgr_498f4d6b_16.03.30_11.35.53.3975558083781694628181.txt com.ibm.ws.orbimpl.transport.WSTransport.createListener 724

[30/03/16 11:35:53:420 BST] 00000001 WsServerImpl  E   WSVR0009E: Error occurred during startup com.ibm.ws.exception.RuntimeError: org.omg.CORBA.INTERNAL: CREATE_LISTENER_FAILED_4  vmcid: 0x49421000  minor code: 56  completed: No

whilst starting a WAS ND Deployment Manager.

Coincidentally (!) this happened RIGHT after I'd upgraded from WAS 8.5.5.4 to 8.5.5.8.

A quick Google search led me here: -

Failure to start dmgr after upgrading BPM/WAS

which said, in part: -

The problem can be related to unrestricted policy file due to SDK upgrade. Please download/install unrestricted policy files, the steps can be found at: http://www14.software.ibm.com/webapp/wsbroker/redirect?version=phil&product=was-nd-dist&topic=tsecegs (Under "Complete the following steps to download and install the new policy files:").

at which point I metaphorically slapped my forehead and said "Doh!".

I fired up my trusty CipherTest Java class: -

java -cp /mnt/Utilities/ CipherTest

which reported: -

FAILED: Max AES key length too small! (128).

I ran my script to update the Java policies to use the unrestricted world-wide ciphers: -

#!/bin/bash

cd /opt/ibm/WebSphere/AppServer/java/jre/lib/security
mv local_policy.jar local_policy.raj
mv US_export_policy.jar US_export_policy.raj
/opt/ibm/WebSphere/AppServer/java/bin/jar xvf /mnt/Java/unrestrictedpolicyfiles.zip
chmod a+rwx *.jar

and re-ran the CipherTest: -

java -cp /mnt/Utilities/ CipherTest

PASSED: Max AES key length OK! - >= 256 (2147483647).

Sorted :-) And "Doh!" And "SLAP!"

Thycotic - Good insight into SSL/TLS

I found this: -

SSL: Beyond the Basics
SSL: Beyond the Basics Part 2: Ciphers
SSL: Beyond the Basics Part 3: Certificates
SSL: Beyond the Basics Part 4: Strict Transport Security

whilst doing some research into the SSL/TLS ciphers supported by various versions of Windows.

Whilst I was specifically reading Part 2, the complete set of articles is definitely something to which I'm going to return and return.

Nice job, Thycotic.

Using OpenSSL to generate personal certificates on Mac OS X and Windows

Following on from a previous post: -

IBM HTTP Server - Tinkering with SSL Client Authentication

and, from my ongoing voyage of discovery this weekend: -

Mac

Generate Private Key

openssl genrsa -out ~/macintosh.uk.ibm.com.key 2048

Create Certificate Service Request

openssl req -new -sha256 -key ~/macintosh.uk.ibm.com.key -out ~/macintosh.uk.ibm.com.csr

Generate Personal Certificate using Certificate Service Request ( on Windows Server 2008 R2 )

certreq.exe -submit -attrib "CertificateTemplate:User" macintosh.uk.ibm.com.csr macintosh.uk.ibm.com.cer

Combine Private Key, Personal Certificate and Certificate Service Request into PKCS12 Export File ( .pfx )

openssl pkcs12 -export -out macintosh.uk.ibm.com.pfx -inkey macintosh.uk.ibm.com.key -in macintosh.uk.ibm.com.cer -certfile macintosh.uk.ibm.com.csr 

The .pfx file can then be imported into a browser such as Firefox, which expects an import to (a) include the private key (b) include the personal certificate and (c) have a password set on the import file

Windows

Setup Path to OpenSSL Configuration

set OPENSSL_CONF=c:\OpenSSL-Win32\bin\openssl.cfg

Generate Private Key

c:\OpenSSL-Win32\bin\openssl.exe genrsa -out c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.key 2048

Create Certificate Service Request

c:\OpenSSL-Win32\bin\openssl.exe req -new -sha256 -key c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.key -out c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.csr

Generate Personal Certificate using Certificate Service Request ( on Windows Server 2008 R2 )

certreq.exe -submit -attrib "CertificateTemplate:User" c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.csr c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.cer

Combine Private Key, Personal Certificate and Certificate Service Request into PKCS12 Export File ( .pfx )

c:\OpenSSL-Win32\bin\openssl.exe pkcs12 -export -out c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.pfx -inkey c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.key -in c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.cer -certfile c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.csr 

- Note; this fails with unable to load certificates - it appears that, with Windows, one does NOT need to provide the CSR, merely the private key and personal certificate

c:\OpenSSL-Win32\bin\openssl.exe pkcs12 -export -out c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.pfx -in c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.cer -inkey c:\Users\Administrator\Desktop\w2k8.uk.ibm.com.key

For the record, on the Mac, I have this version of OpenSSL: -

openssl version

OpenSSL 0.9.8zh 14 Jan 2016

c:\OpenSSL-Win32\bin\openssl.exe version

OpenSSL 1.0.2g  1 Mar 2016

"javax.net.ssl.SSLHandshakeException: no cipher suites in common" seen in WebSphere Application Server Node Agent logs

Following hot on the heels from my last post: -

Error handshake_failure seen when connecting to WebSphere Application Server using SOAP over HTTPS

I started seeing: -

…

[26/03/16 15:38:45:206 GMT] 00000048 SSLHandshakeE E   SSLC0008E: Unable to initialize SSL connection.  Unauthorized access was denied or security settings have expired.  Exception is javax.net.ssl.SSLHandshakeException: no cipher suites in common
…

Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common
…

[26/03/16 15:41:13:771 GMT] 00000075 ORBRas        E com.ibm.ws.security.orbssl.WSSSLClientSocketFactoryImpl createSSLSocket ProcessDiscovery : 0 JSSL0080E: javax.net.ssl.SSLHandshakeException - The client and server could not negotiate the desired level of security.  Reason: Received fatal alert: handshake_failure javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
…

in the SystemOut.log of the Node Agent.

The solution was, thankfully, the same; I updated the node-level ssl.client.props : -

vi /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/properties/ssl.client.props 

and adding: -

com.ibm.ssl.enabledCipherSuites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Error handshake_failure seen when connecting to WebSphere Application Server using SOAP over HTTPS

I saw a bunch of SSL-related errors when attempting to use / access the WebSphere Application Server SOAP-based administration service: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd -host `hostname` -port 8879

including: -

...

WASX7023E: Error creating "SOAP" connection to host "bpm856.uk.ibm.com"; exception information: com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure; targetException=java.lang.IllegalArgumentException: Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure]

…

com.ibm.websphere.management.exception.ConnectorNotAvailableException: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure; targetException=java.lang.IllegalArgumentException: Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure]
...

Caused by: [SOAPException: faultCode=SOAP-ENV:Client; msg=Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure; targetException=java.lang.IllegalArgumentException: Error opening socket: java.io.IOException: Exception during sslSocket.startHandshake: Received fatal alert: handshake_failure]…

This started happening immediately after I'd locked down WAS using TLS 1.2 and Mutual Authentication.

However, all had been working UNTIL I enforced WAS to use a pair of TLS 1.2 ciphers: -

Specifically, I'm using these: -

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

across the "estate" including IBM HTTP Server, WAS and DB2.

This proved to be the root cause ….

I needed to update the Deployment Manager profile's SSL configuration: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/properties/ssl.client.props

from: -

#com.ibm.ssl.enabledCipherSuites=

to: -

com.ibm.ssl.enabledCipherSuites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

*NOTE* The list is separated with a space character NOT a comma; see: -

ssl.client.props client configuration file

for details.

For the record, I'd previously done this when I enforced TLS 1.2, by changing: -

com.ibm.ssl.protocol=SSL_TLS

to: -

com.ibm.ssl.protocol=TLSv1.2

IBM HTTP Server - Tinkering with SSL Client Authentication

I'm tinkering with so-called Mutual Authentication (MA) between various middleware components, from browser through to back-end WebSphere Application Server (WAS).

Here's what I'm reading: -

Client authentication

IBM HTTP SSL Server Questions and Answers

SSL Directives

I'm also referencing this: -

The Most Common OpenSSL Commands

as I'm using openSSL to create personal device certificates etc. to allow a client browser to MA to IHS: -

Generate a Private Key on my Mac

openssl genrsa -out ~/macintosh.uk.ibm.com.key 2048

Generate a Certificate Service Request

openssl req -new -sha256 -key ~/macintosh.uk.ibm.com.key -out ~/macintosh.uk.ibm.com.csr

Generate a Personal Certificate from the Certificate Service Request

certreq -submit -attrib "CertificateTemplate:Webserver" macintosh.uk.ibm.com.csr macintosh.uk.ibm.com.cer

- This on a Windows Server 2008 R2 box

Convert the Personal Certificate and Private Key and Certificate Service Request into a PKCS12 file

openssl pkcs12 -export -out macintosh.uk.ibm.com.pfx -inkey macintosh.uk.ibm.com.key -in macintosh.uk.ibm.com.cer -certfile macintosh.uk.ibm.com.csr 

- This is required to allow me to import the private key and personal certificate into Firefox

For the record, this is relevant part of my IHS configuration - httpf.conf

…

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 8443
<VirtualHost *:8443>
SSLProtocolDisable SSLv2 SSLv3 TLSv10 TLSv11
SSLProtocolEnable TLSv12
SSLCipherSpec ALL NONE
SSLCipherSpec TLSv12 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
SSLCipherSpec TLSv12 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
SSLClientAuth Required
SSLEnable
</VirtualHost>
KeyFile /opt/IBM/HTTPServer/ssl/keystore.kdb
SSLDisable

...

and this is the validation of same: -

/opt/IBM/HTTPServer/bin/apachectl -DDUMP_SSL_CONFIG

SSL configuration:
Default server
Server name: bpm856.uk.ibm.com:8080
SSL enabled: NO

SSL server defined at: /opt/IBM/HTTPServer/conf/httpd.conf:852
Server name: bpm856.uk.ibm.com:8443
SSL enabled: YES
FIPS enabled: 0
Keyfile: /opt/IBM/HTTPServer/ssl/keystore.kdb
Protocols enabled: TLSv12
Ciphers for SSLV2: (protocol disabled)
Ciphers for SSLV3: (protocol disabled)
Ciphers for TLSv10: (protocol disabled)
Ciphers for TLSv11: (protocol disabled)
Ciphers for TLSv12: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(C02F),TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384(C030)

Syntax OK

and this is how I validate IHS from a client connection, using openSSL: -

openssl s_client -connect bpm856.uk.ibm.com:8443 < /dev/null

…

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
…

Java EE, the next inception: A primer to WebSphere Liberty for Java EE developers

I saw this on Twitter and wanted to reshare: -

Java EE, the next inception: A primer to WebSphere Liberty for Java EE developers

Learn the basics of how IBM® WebSphere® Application Server Liberty profile works, and how its architecture differs significantly from that of older Java™ EE application servers. This background will be helpful for understanding how to install a local Java EE application development environment using Eclipse and Liberty, as well as how to deploy Liberty applications.

Something else with which to play ….