Tuesday, 13 October 2015

WebSphere Application Server - Tinkering with Property Files and a spot of SSL/TLS

This is in the context of my ongoing voyage of discovery with Transport Layer Security (TLS) 1.2 within a WebSphere Application Server (WAS) 8.5.5.7 cell.

It is, for this reason, that I was reading this: -


Specifically, having enabled TLS 1.2 as the default protocol for my cell: -

/opt/ibm/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

cellID=AdminControl.getCell()

AdminTask.modifySSLConfig('[-alias CellDefaultSSLSettings -scopeName (cell):'+cellID+' -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):'+cellID+' -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):'+cellID+' -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers ]')

As soon as I save and synchronise the changes: -

AdminConfig.save()
AdminNodeManagement.syncActiveNodes()


I see this exception: -

ADMS0206I: The configuration synchronization failed for node: AppSrv01Node.

in the Deployment Manager's SystemOut.log file, and this: -

[08/10/15 15:19:45:430 BST]     FFDC Exception:javax.net.ssl.SSLHandshakeException SourceId:com.ibm.ws.management.filetransfer.client.FileTransferClientImpl.download ProbeId:1575 Reporter:com.ibm.ws.management.filetransfer.client.FileTransferClientImpl@a36b36a1
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake

...

Caused by: java.io.EOFException: SSL peer shut down incorrectly

in the Node Agent's FFDC log.

"Talking" this through with the wider WebSphere team, it was pointed out that, by default, I have Dynamic SSL Configuration enabled: -

which means that, as soon as I committed the change, the Deployment Manager immediately starting using AND enforcing TLS 1.2, whereas the Node Agent(s) continued using SSL/TLS.

I could see this by comparing/contrasting the cell and node-level security.xml files: -

diff AppSrv01/config/cells/WASCell/security.xml Dmgr01/config/cells/WASCell/security.xml

108c108
<     <setting xmi:id="SecureSocketLayer_1" clientAuthentication="false" securityLevel="HIGH" enabledCiphers="" jsseProvider="IBMJSSE2" sslProtocol="SSL_TLS" keyStore="KeyStore_1" trustStore="KeyStore_2" trustManager="TrustManager_2" keyManager="KeyManager_1"/>
---
>     <setting xmi:id="SecureSocketLayer_1" clientAuthentication="false" securityLevel="HIGH" enabledCiphers="" jsseProvider="IBMJSSE2" sslProtocol="TLSv1.2" keyStore="KeyStore_1" trustStore="KeyStore_2" trustManager="TrustManager_2" keyManager="KeyManager_1"/>

where the first is from the node and the second is from the DM.

Bill's presentation ( see the earlier link ) explicitly warns about this, on page 56: -

WARNING: BE SURE TO DISABLE dynamic SSL Prior to turning on strict mode. The Dmgr and Nodes will be in a incompatible mode. The Dmgr will likely switch to TLSv1.2 before the nodes do, they will likely still be at SSL_TLS, and the nodes and dmgr will no long be able to communicate. We STRONGLY RECOMMEND that you stop all the nodes except the Dmgr. Do the conversion on the console, restart the Dmgr, manually sync the nodes, then start the node agents and servers.

To start with, I experimented by disabling Dynamic SSL Configuration.

Whilst I knew that I could manually modify security.xml I was looking for a programmatic / scripted way of achieving this.

This is what I did: -

configProperties=AdminConfig.list('Cell')
AdminTask.extractConfigProperties(configProperties, '[-propertiesFileName /tmp/davehay.properties -filterMechanism SELECTED_SUBTYPES -selectedSubTypes [Security ] ]')


The resulting file - /tmp/davehay.properties - includes the line: -

dynamicallyUpdateSSLConfig=true #boolean,default(false)

Therefore, I created a pair of property files: -

dynamic_off.properties

# SubSection 1.0 # Security Section
#
ResourceType=Security
ImplementingResourceType=Security
ResourceId=Cell=!{cellName}:Security=ID#Security_1
#

#
#Properties
#
dynamicallyUpdateSSLConfig=false #boolean,default(false)


dynamic_on.properties

# SubSection 1.0 # Security Section
#
ResourceType=Security
ImplementingResourceType=Security
ResourceId=Cell=!{cellName}:Security=ID#Security_1
#

#
#Properties
#
dynamicallyUpdateSSLConfig=true #boolean,default(false)


which allowed me to turn off Dynamic Security Configuration: _

AdminTask.validateConfigProperties('-propertiesFileName /tmp/dynamic_off.properties')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

which returns 'true'

and then turn it back on again: -

AdminTask.validateConfigProperties('-propertiesFileName /tmp/dynamic_off.properties')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()


which again returns 'true' :-)

However, I realised that I was going against the recommendation from Bill i.e. to ONLY have the Deployment Manager running, make the SSL_TLS > TLSv1.2 change and then manually synchronise the node(s).

This is what I did, in summary: -

------------------------------------------------------------------------------------

(a) Fully syncronise the cell

(b) Stop the Node Agent

(c) Modify the SSL configuration: -

AdminTask.modifySSLConfig('[-alias CellDefaultSSLSettings -scopeName (cell):WASCell -keyStoreName CellDefaultKeyStore -keyStoreScopeName (cell):WASCell -trustStoreName CellDefaultTrustStore -trustStoreScopeName (cell):WASCell -jsseProvider IBMJSSE2 -sslProtocol TLSv1.2 -clientAuthentication false -clientAuthenticationSupported false -securityLevel HIGH -enabledCiphers ]')

(d) Update the Node's ssl.client.props file to reflect the fact that we're now using TLS v1.2 rather than SSL_TLS

vi /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/properties/ssl.client.props

changing from: -

com.ibm.ssl.protocol=SSL_TLS

to: -

com.ibm.ssl.protocol=TLSv1.2

(e) Synchronise the Node with the cell: -

 /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/bin/syncNode.sh `hostname` -user wasadmin -password passw0rd

(f) Validate that DM and Node have consistent SSL/TLS settings: -

diff /opt/ibm/WebSphere/AppServer/profiles/Dmgr01/config/cells/WASCell/security.xml /opt/ibm/WebSphere/AppServer/profiles/AppSrv01/config/cells/WASCell/security.xml

(g) Start the node

(h) Validate that I can see the node from the DM

(i) Mix a Margherita     

------------------------------------------------------------------------------------

So, what did I learn ?

Firstly, read Bill's document from cover to cover.
Secondly, remember Dynamic SSL Configuration.
Thirdly, I now know how to work with WAS property files in Jython
Fourthly, I like Margheritas

No comments:

Note to self - use kubectl to query images in a pod or deployment

In both cases, we use JSON ... For a deployment, we can do this: - kubectl get deployment foobar --namespace snafu --output jsonpath="{...