Saturday 23 April 2016

WebSphere Application Server - Converting Internal CA Certificates from one Signature Algorithm to Another

I have a need to switch my internal WAS cell-default certificates ( root, default etc. ) from one Signature Algorithm ( SHA1 ) to another ( SHA256 )

We have an app ^H^H^H tool for that.

Here's my journey: -

Start WSAdmin

/opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

Convert Certificates to SHA256withRSA

AdminTask.convertCertForSecurityStandard('[-fipsLevel SP800-131 -signatureAlgorithm SHA256withRSA -keySize 2048 ]')

Save and Sync

AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Quit

quit

PS Next time one starts WSAdmin, updated certificates are retrieved back to "local" trust store

/opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

*** SSL SIGNER EXCHANGE PROMPT ***
SSL signer from target host 127.0.0.1 is not found in trust store /opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/etc/trust.p12.

Here is the signer information (verify the digest value matches what is displayed at the server): 

Subject DN:    CN=bpm857.uk.ibm.com, OU=PCCell1, OU=Dmgr, O=IBM, C=US
Issuer DN:     CN=bpm857.uk.ibm.com, OU=Root Certificate, OU=PCCell1, OU=Dmgr, O=IBM, C=US
Serial number: 23501208165426
Expires:       Thu Apr 20 05:51:34 BST 2017
SHA-1 Digest:  AA:C8:35:CC:B3:46:2E:CE:E5:05:01:A7:5B:55:3A:DF:3C:06:44:A0
MD5 Digest:    76:CA:9D:33:0B:41:EA:03:F9:7E:A6:C6:02:65:D7:4D

Subject DN:    CN=bpm857.uk.ibm.com, OU=Root Certificate, OU=PCCell1, OU=Dmgr, O=IBM, C=US
Issuer DN:     CN=bpm857.uk.ibm.com, OU=Root Certificate, OU=PCCell1, OU=Dmgr, O=IBM, C=US
Serial number: 23500614165097
Expires:       Thu Apr 17 05:51:33 BST 2031
SHA-1 Digest:  AA:C8:35:CC:B3:46:2E:CE:E5:05:01:A7:5B:55:3A:DF:3C:06:44:A0
MD5 Digest:    76:CA:9D:33:0B:41:EA:03:F9:7E:A6:C6:02:65:D7:4D

Add signer to the trust store now? (y/n) y

Validate DM Profile Personal Certificate

/opt/IBM/WebSphere/AppServer/java/jre/bin/ikeycmd -cert -list -db /opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/config/cells/PCCell1/key.p12 -pw WebAS

Certificates in database /opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/config/cells/PCCell1/key.p12:
   default

/opt/IBM/WebSphere/AppServer/java/jre/bin/ikeycmd -cert -details -db /opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/config/cells/PCCell1/key.p12 -pw WebAS -label default

...
Label: default
Key Size: 2048
...
Signature Algorithm: SHA256withRSA (1.2.840.113549.1.1.11)
...

Validate DM Profile Root Certificate

/opt/IBM/WebSphere/AppServer/java/jre/bin/ikeycmd -cert -list -db /opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/config/cells/PCCell1/trust.p12 -pw WebAS

Certificates in database /opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/config/cells/PCCell1/trust.p12:
   root

/opt/IBM/WebSphere/AppServer/java/jre/bin/ikeycmd -cert -details -db /opt/IBM/WebSphere/AppServer/profiles/PCDmgr01/config/cells/PCCell1/trust.p12 -pw WebAS -label root

...
Label: root
Key Size: 2048
...
Signature Algorithm: SHA256withRSA (1.2.840.113549.1.1.11)
...

Manually synchronise Node - this may be optional but felt "belt n' braces"

/opt/IBM/WebSphere/AppServer/profiles/PCAppSrv01/bin/syncNode.sh `hostname` 10003

...
*** SSL SIGNER EXCHANGE PROMPT ***
SSL signer from target host 127.0.0.1 is not found in trust store /opt/IBM/WebSphere/AppServer/profiles/PCAppSrv01/etc/trust.p12.

Here is the signer information (verify the digest value matches what is displayed at the server): 

Subject DN:    CN=bpm857.uk.ibm.com, OU=PCCell1, OU=Dmgr, O=IBM, C=US
Issuer DN:     CN=bpm857.uk.ibm.com, OU=Root Certificate, OU=PCCell1, OU=Dmgr, O=IBM, C=US
Serial number: 35849612167890
Expires:       Sat Apr 22 07:14:50 BST 2017
SHA-1 Digest:  94:26:35:50:26:01:43:84:9B:68:63:C8:48:1B:E6:CF:87:E6:85:18
MD5 Digest:    7A:E0:99:2B:CB:65:E7:09:63:00:E9:63:00:9B:84:E8

Subject DN:    CN=bpm857.uk.ibm.com, OU=Root Certificate, OU=PCCell1, OU=Dmgr, O=IBM, C=US
Issuer DN:     CN=bpm857.uk.ibm.com, OU=Root Certificate, OU=PCCell1, OU=Dmgr, O=IBM, C=US
Serial number: 35848947653836
Expires:       Sat Apr 19 07:14:49 BST 2031
SHA-1 Digest:  94:26:35:50:26:01:43:84:9B:68:63:C8:48:1B:E6:CF:87:E6:85:18
MD5 Digest:    7A:E0:99:2B:CB:65:E7:09:63:00:E9:63:00:9B:84:E8

Add signer to the trust store now? (y/n) y
...

Validate Node Profile Personal Certificate

/opt/IBM/WebSphere/AppServer/java/jre/bin/ikeycmd -cert -list -db /opt/IBM/WebSphere/AppServer/profiles/PCAppSrv01/config/cells/PCCell1/key.p12 -pw WebAS

Certificates in database /opt/IBM/WebSphere/AppServer/profiles/PCAppSrv01/config/cells/PCCell1/key.p12:
   default

/opt/IBM/WebSphere/AppServer/java/jre/bin/ikeycmd -cert -details -db /opt/IBM/WebSphere/AppServer/profiles/PCAppSrv01/config/cells/PCCell1/key.p12 -pw WebAS -label default

...
Label: default
Key Size: 2048
...
Signature Algorithm: SHA256withRSA (1.2.840.113549.1.1.11)
...

Validate Node Profile Root Certificate

/opt/IBM/WebSphere/AppServer/java/jre/bin/ikeycmd -cert -list -db /opt/IBM/WebSphere/AppServer/profiles/PCAppSrv01/config/cells/PCCell1/trust.p12 -pw WebAS

Certificates in database /opt/IBM/WebSphere/AppServer/profiles/PCAppSrv01/config/cells/PCCell1/trust.p12:
   root

/opt/IBM/WebSphere/AppServer/java/jre/bin/ikeycmd -cert -details -db /opt/IBM/WebSphere/AppServer/profiles/PCAppSrv01/config/cells/PCCell1/trust.p12 -pw WebAS -label root

...
Label: root
Key Size: 2048
...
Signature Algorithm: SHA256withRSA (1.2.840.113549.1.1.11)
...

Additional Validation using OpenSSL

echo "" | openssl s_client -connect bpm857.uk.ibm.com:10001 -prexit 2>/dev/null | openssl x509 -noout -text | grep Signature

...
    Signature Algorithm: sha256WithRSAEncryption
    Signature Algorithm: sha256WithRSAEncryption
...

So that's all good then.

Friday 15 April 2016

WebSphere User Group, IBM South Bank, Monday 25 April 2016

We are pleased to confirm that the next meeting of the WebSphere User Group (UK) will take place on Monday 25th April 2016, to be held in the IBM Client Centre at IBM South Bank, London. There will be no charge for this meeting and as usual we aim to have an excellent agenda lined up, covering a range of current and emerging WebSphere technologies.

Provisional Agenda


For the most recent agenda, logistics and registration, please visit the WUG website here.

Hope to see you there - I'll be speaking about DevOps and BPM, which is nice.

Wednesday 13 April 2016

MacBook Pro and USB - Interesting Quirkiness

I'm using a MacBook Pro: -


I've seen this a few times: -

034378.680514 SSP4@14700000: AppleUSB30XHCIPort::resetAndCreateDevice: failed to create device after (1) tries, disabling port
034414.624087 SSP4@14700000: AppleUSB30XHCIPort::resetAndCreateDevice: failed to create device after (1) tries, disabling port
034416.245827 SSP4@14700000: AppleUSB30XHCIPort::resetAndCreateDevice: failed to create device after (1) tries, disabling port
037923.490801 SSP1@14500000: AppleUSB30XHCIPort::resetAndCreateDevice: failed to create device after (1) tries, disabling port
037961.216945 SSP2@14600000: AppleUSB30XHCIPort::resetAndCreateDevice: failed to create device after (1) tries, disabling port
038140.039979 SSP2@14600000: AppleUSB30XHCIPort::resetAndCreateDevice: failed to create device after (1) tries, disabling port


when I insert a USB 2.0 or USB 3.0 hard drive.

I only started looking in the system log ( sudo dmesg ) when the USB drives failed to materialise in Finder or Disk Utility.

Previously I'd "solved" this by rebooting, which is a pain.

For the record, I also saw the same on a Mac Mini


This time around, I had a quick Google and found this: -


which said, in part: -

All Macs with USB3 ports are UASP capable and UAS in enabled by default. Since Mountain Lion 10.8 Apple has included kernel extensions (kext) for UAS. In System>Library>Extensions>IOUSBAttachedSCSI.kext (Apple wouldn't include the kext if Macs didn't support UASP)
So yes, Macs do, by default, I might add, use UASP when a UASP enabled device is attached using a USB3 cable. If a UAS device is not detected, then Macs revert to BOT. This has been true since at least the rMBP released in 2012. (Which is what I'm using right now)
...
On occasion, UAS will not be used due to USB hubs, USB2 devices being inserted into the ports (shared bus) and when the USB3 port gets stuck in BOT protocol. Always best to attach a USB3 UAS device first into the USB 3 port. If the Mac gets stuck on USB2 or BOT, removing all devices and a reboot normally clears it.

Taking this into account, I tested by plugging the USB drive into a USB 3.0 hub: -


This time around the drive just worked ….

… which is nice

Weird but nice :-)

WebSphere Application Server - Managing the Service Integration Bus using Python

A friend asked me: -

Hi Dave -- do you know the AdminConfig.getid(type:scope) to get a SIBus engine? 
not sure what type and scope need to be passed in 

It took me a wee while, but I shared this: -

for bus in AdminTask.listSIBuses().splitlines():
 name=AdminConfig.showAttribute(bus, "name")
 print AdminConfig.getid("/SIBus:"+ name)                                                                         

which results in: -

BPM.PCCellDe1.Bus(cells/PCCell1/buses/BPM.PCCellDe1.Bus|sib-bus.xml#SIBus_1460402518228)

which is nice.

Tuesday 12 April 2016

IBM WebSphere Plugin - GSK_ERROR_BAD_CERT and GSK_INVALID_HANDLE after an upgrade

During a recent transition from SHA1 to SHA2 signature algorithms *AND* an upgrade from WebSphere Application Server (WAS) 8.5.5.4 to 8.5.5.8, we hit an interesting challenge yesterday.

We're using IBM HTTP Server (IHS) and the WebSphere Plugin on one AIX LPAR, fronting IBM Integration Bus (IIB) on another LPAR.

We've got a set of IIB flows, all of which are being offered up via IHS through the WebSphere Plugin configuration.

Once the 8.5.5.8 upgrade ( including IHS and Plugin ) was completed, we started seeing GSK_ERROR_BAD_CERT and GSK_INVALID_HANDLE errors in the Plugin error log, relating to the downstream IIB HTTP listeners: -

We're still debugging this BUT it looks like the 8.5.5.8 introduced a security validation check, as per this: -


( Actually, this was introduced in 8.5.5.7 )

We appear to have a "problem" with one of our signer certificates, in terms of a mismatch against standards, and the Plugin is now picking up on this.

( Remember the Plugin is the client to IIB, so this is all about that connection, rather than the connection TO IHS )

Thankfully, we do have a mitigation; by setting the Config parameter AutoSecurity="false" in the Plugin configuration file, the security checking is disabled (!), meaning that we can now client from Plugin to IIB.

Now we need to go and revisit our signer certificates ….. but at least we can continue to test ( this is a NON-PRODUCTION environment )

Friday 8 April 2016

IBM Business Monitor - CWMAX4203E: RESTSecurityAdminMBean instance was not found

I saw this error earlier: -

CWMAX4203E: RESTSecurityAdminMBean instance was not found.

whilst checking my Monitor models, in IBM Business Monitor 8.5.5, after making some SSL/TLS related changes between WebSphere Application Server (WAS) and DB2.

I saw this when I navigated to Applications > Monitor Models.

This caused me to briefly panic.

Thankfully, I found that someone had seen a similar issue, documented in a Problem Management Report (PMR).

For context, in order to make my configuration changes, I needed to have the Deployment Manager running, in order to import two new SSL signer certificates and also select two new, specific, TLS ciphers.

Therefore, I was changing the Deployment Manager configuration, before manually synchronising the node agents.

This meant that, briefly, the DM tried and failed to connect to DB2.

I suspect that this was root cause - from reading the PMR, I believe that a WAS component, LifecycleServicesStartup, starts RESTSecurityAdminMBean during the server startup.

Long story short, I restarted the Deployment Manager, and the problem went away.

I did double-check that: -

(a) I could access my list of Monitor models
(b) click into any specific Model, and see a nice set of green ticks

(c) generate a set of updated Cognos cubes for the Model


which is nice

Wednesday 6 April 2016

Book Review - The Purpose of Change is Problem Solving

This is the latest in my series of relatively infrequent book reviews for the British Computer Society: -

I chose this book mainly based upon the title, to which I related as an
inveterate solver of problems.

Whilst the book is absolutely about problem solving, it wasn't quite
what I expected. In this book, the author, Janos Korn, digs deeply into
the use of language, including semantics, natural language and
structure.

Thus I found the book to be of somewhat limited relevance to my
day-to-day work as an engineer, IT consultant, problem solver etc.

By this, I don't mean to imply that the book isn't likely to be of use
to many others, I found the language, repeated and multiple references
and, most importantly, the format to be of limited interest.

With specific regard to formatting, I'd have preferred the book to make
more use of indentation and punctuation, as it was hard to separate the
paragraphs from the examples.

So, in summary, whilst this is obviously an extremely well-written and
deeply researched publication, I found it to be a somewhat arduous and
less-than-relevant read.

Given that I obviously failed to understand the purpose and target
audience of this book, I cannot find major fault with the publication
itself, but, in terms of relevance to me, the reader, I'd give it 6/10.


For the record, I was kindly provided this book by BCS, at no cost to myself.

Tuesday 5 April 2016

My Cognos has fallen and can't get up

Purely FYI, in case you hit this problem, someone (!) managed to break my IBM Business Monitor 8.5.5 installation over the weekend, most likely when the underlying AIX LPARs were shut down and moved from one physical box to another.

I saw a bunch of nasty exceptions in the Cognos instance pogo logs: -

 2016-04-05 09:21:54.467 FATAL [.authorization.AuthorizationAdapterFactory] Thread-95: Unable to initialize the Access Control Module
com.ibm.cognos.internal.camaaa.accesscontrol.AccessControlException: AAA-ACM-0011 Failed to create http client due to CAMCryptoExce
ption
 
Caused by: com.ibm.cognos.camaaa.internal.common.exception.LocalizableException: AAA-CFG-0016 CAM Crypto initialization failed. Plea
se verify the cryptographic configuration settings

Caused by: CAM-CRP-1280 An error occurred while trying to decrypt using the system protection key. Reason: javax.crypto.BadPaddingEx
ception: Given final block not properly padded 


which threw me somewhat.

Rather than panicking, I read a bunch of Technotes and PMRs, and then decided to simply blow away the Cognos configuration that's stored on each of the WAS app nodes ( under the WAS profiles directory ).

I restarted the cluster, one JVM at a time ( TWICE to allow the configuration to be properly rebuilt ) and all now appears well.

I did have to go back into each of the two JVMs and manually update cogstartup.xml to reflect the correct DB2 listener port ( we've moved to TLS connections between WAS and DB2 for everything apart from Cognos ), and again restart the cluster members.

But all now appears well :-)

If I had to bet, I'd guess that there's something unique in the Cognos configuration, in terms of encryption keys ( see above messages ), perhaps where the key is based upon something unique to the underlying hardware platform ( I remember reading about how AIX and AES ciphers work ).

Therefore, the configuration had the OLD keys for the OLD hardware, whereas we've moved the LPARs to NEW hardware.

Visual Studio Code - Wow 🙀

Why did I not know that I can merely hit [cmd] [p]  to bring up a search box allowing me to search my project e.g. a repo cloned from GitHub...