Wednesday, 10 April 2013

CWWIM1998E The following system exception occurred during processing: 'java.text.ParseException: Unparseable date: "20120803122609.732-0000Z"'

Now I cannot take the credit for resolving this one, but I wanted to share the problem and a potential resolution.

Having completed the implementation of IBM Business Monitor 8.0.1.1 for a client, we found, during Functional Testing, that we were unable to log into BusinessSpace: -


using a LDAP user account.

This failed with an HTTP 500 ( which normally indicates a very serious problem with WAS ).

When we checked SystemOut.log, we found

[09/04/13 11:37:36:558 BST] 00000029 exception     W com.ibm.ws.wim.adapter.ldap.LdapAdapter getDateString(Object) CWWIM1998E  The following system exception occurred during processing: 'java.text.ParseException: Unparseable date: "20120803122609.732-0000Z"'.
[09/04/13 11:37:36:559 BST] 00000029 exception     W com.ibm.ws.wim.adapter.ldap.LdapAdapter getDateString(Object)
                                 com.ibm.websphere.wim.exception.WIMSystemException: CWWIM1998E  The following system exception occurred during processing: 'java.text.ParseException: Unparseable date: "20120803122609.732-0000Z"'.

[09/04/13 11:37:36:566 BST] 00000029 servlet       E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: An exception was thrown by one of the service methods of the servlet [mmOSGI] in application [mm.was_bamweb-cluster-80-tatc-i-1]. Exception created : [java.lang.RuntimeException: com.ibm.mm.server.model.user.exception.CannotLoadUserException: BMWSM0016E: The user data could not be loaded.

The symptoms appeared to be a match for: -

We do have a custom LDAP configured ( we're using CA LDAP for which WAS doesn't have a template, as it does for ITDS, AD, Domino etc. ).

It's worth noting that CA LDAP is supported as long as it complies with the LDAP v3 specification, which it does.

The APAR also has a circumvention; to add the offending date format to WIM ( wimconfig.xml ): -

One will have to run following command in order to support
specified dateFormat by VMM.
$AdminTask setIdMgrCustomProperty {-id <LDAP_REPO_ID> -name
ldapTimestampFormat  -value "yyyyMMddHHmmssZ" }
yyyyMMddHHmmssZ >> may vary as per LDAP's dateFormat!

Given that the dates we were getting were in this format: -

20120803122609.732-0000Z

we tried a number of variations on the ldapTimestampFormat property: -

$ /opt/ibm/WebSphere/AppServer/profiles/dm-80-tatc-i/bin/wsadmin.sh -lang jython
wsadmin> AdminTask.setIdMgrCustomProperty('-id ldpa -name ldapTimestampFormat  -value "yyyyMMddHHmmss.Z"')
wsadmin> AdminConfig.save()

-or-

wsadmin> AdminTask.setIdMgrCustomProperty('-id ldpa -name ldapTimestampFormat  -value "yyyyMMddHHmmss.SZ"')
wsadmin> AdminConfig.save()

-or-

wsadmin> AdminTask.setIdMgrCustomProperty('-id ldpa -name ldapTimestampFormat  -value "yyyyMMddHHmmss.SSS-SSSSZ"')
wsadmin> AdminConfig.save()

In each case, we noted that a new line was added / modified in wimconfig.xml: -

      <config:CustomProperties name="ldapTimestampFormat" value="yyyyMMddHHmmss.Z"/>

However, after synchronising the nodes and restarting the JVM hosting BusinessSpace ( the WebApp cluster ), we still saw the same exception.

At that point we were baffled.

Thankfully, the client had previously seen this problem with WebSphere Portal on an earlier version of WAS, and had been recommended to set two different properties in wimconfig.xml to mark two particular LDAP attributes as being unsupported.

To achieve this, we removed the previously added ldapTimestampFormat property: -

$ AdminTask.setIdMgrCustomProperty('-id ldpa -name ldapTimestampFormat')
$ AdminConfig.save()

and then tried to add the unsupported attribute entries: -

wsadmin> AdminTask.addIdMgrLDAPAttrNotSupported('-id ldpa -propertyName modifyTimestamp')
wsadmin> AdminTask.addIdMgrLDAPAttrNotSupported('-id ldpa -propertyName createTimestamp')
wsadmin> AdminConfig.save()

Sadly this failed with a NullPointerException.

Looking at this APAR: -

PM75085: THE WSADMIN ADDIDMGRLDAPATTRNOTSUPPORTED CLI RETURNS NULLPOINTEREXCEPTION FOR VMM CUSTOM REPOSITORY

it appears that this is a known issue, which is due to be fixed in WAS 8.0.0.6 :-)

However, we were able to resolve the issue, by manually adding the attributeConfiguration block, with reference to the two unsupported attributes, to wimconfig.xml: -

Old

      <config:ldapEntityTypes name="Group" searchFilter="">
        <config:objectClasses>groupOfNames</config:objectClasses>
        <config:searchBases>ou=users,o=ibm</config:searchBases>
      </config:ldapEntityTypes>
    </config:repositories>


New

      <config:ldapEntityTypes name="Group" searchFilter="">
        <config:objectClasses>groupOfNames</config:objectClasses>
        <config:searchBases>ou=users,o=ibm</config:searchBases>
      </config:ldapEntityTypes>
      <config:attributeConfiguration>
        <config:propertiesNotSupported name="modifyTimestamp"/>
        <config:propertiesNotSupported name="createTimestamp"/>
      </config:attributeConfiguration>

    </config:repositories>

before synchronising the nodes and restarting the JVM.

This time, it worked, and we were able to successfully log into BusinessSpace using an LDAP user.

We are going to raise a PMR against BusinessSpace, as it's not clear why the original problem only occurred for that LDAP, when we were able to log into the WAS Integrated Solutions Console ( and into the Cognos Dispatcher ) using a LDAP user.

Therefore, this MAY be a new issue specifically with BusinessSpace, although I'm struggling to work out why the WAS-based solution ( iFix and/or ldapTimestampFormat property ) did not work, given that BusinessSpace runs on the WAS JVM …….

Thankfully, my client had retained their notes from the previous occurrence of the problem, which saved us this time around. Kudos to Paul and Tony for writing stuff down :-)

Definitely one for the "Hmmmm" pile.

PS There is also an excellent developerWorks article: -


which was of relevance. Whilst it didn't provide the solution ( using config:propertiesNotSupported ), it did provide some good background on the problem.

No comments:

Note to self - use kubectl to query images in a pod or deployment

In both cases, we use JSON ... For a deployment, we can do this: - kubectl get deployment foobar --namespace snafu --output jsonpath="{...