Tuesday, 5 April 2016

My Cognos has fallen and can't get up

Purely FYI, in case you hit this problem, someone (!) managed to break my IBM Business Monitor 8.5.5 installation over the weekend, most likely when the underlying AIX LPARs were shut down and moved from one physical box to another.

I saw a bunch of nasty exceptions in the Cognos instance pogo logs: -

 2016-04-05 09:21:54.467 FATAL [.authorization.AuthorizationAdapterFactory] Thread-95: Unable to initialize the Access Control Module
com.ibm.cognos.internal.camaaa.accesscontrol.AccessControlException: AAA-ACM-0011 Failed to create http client due to CAMCryptoExce
ption
 
Caused by: com.ibm.cognos.camaaa.internal.common.exception.LocalizableException: AAA-CFG-0016 CAM Crypto initialization failed. Plea
se verify the cryptographic configuration settings

Caused by: CAM-CRP-1280 An error occurred while trying to decrypt using the system protection key. Reason: javax.crypto.BadPaddingEx
ception: Given final block not properly padded 


which threw me somewhat.

Rather than panicking, I read a bunch of Technotes and PMRs, and then decided to simply blow away the Cognos configuration that's stored on each of the WAS app nodes ( under the WAS profiles directory ).

I restarted the cluster, one JVM at a time ( TWICE to allow the configuration to be properly rebuilt ) and all now appears well.

I did have to go back into each of the two JVMs and manually update cogstartup.xml to reflect the correct DB2 listener port ( we've moved to TLS connections between WAS and DB2 for everything apart from Cognos ), and again restart the cluster members.

But all now appears well :-)

If I had to bet, I'd guess that there's something unique in the Cognos configuration, in terms of encryption keys ( see above messages ), perhaps where the key is based upon something unique to the underlying hardware platform ( I remember reading about how AIX and AES ciphers work ).

Therefore, the configuration had the OLD keys for the OLD hardware, whereas we've moved the LPARs to NEW hardware.

No comments: