IBM BPM and Oracle - another interesting problem

Earlier this week, I was working with a client to grow their BPM development environment from two to four nodes, meaning that the Deployment Environment effectively doubled in size.

We achieved this by editing the template ( Advanced-PS-ThreeClusters-Oracle.properties ) and adding the additional two nodes, each hosting three new cluster members, and then using the BPMConfig.sh script to update the Deployment Environment as follows: -

./BPMConfig.sh -create -de Advanced-PS-ThreeClusters-Oracle.properties

Having done this, we started up the DE, and validated the changes by hitting the Process Portal, Process Admin and BPC UIs, and also stopping the old nodes ( 1 and 2 ) and ensuring that the service carried on running on nodes 3 and 4.

Apart from briefly forgetting to regenerate/propagate the WebSphere Plugin configuration and then restart IBM HTTP Server, all was well.

However, when we started the core business application ( which is an Enterprise Archive comprising SCA modules with mediations and BPEL flows ), we saw this in the SystemOut.log for all of the AppCluster members, both the original AND the new ones: -

java.lang.NoClassDefFoundError: oracle.xdb.XMLType

Caused by: java.lang.ClassNotFoundException: oracle.xdb.XMLType
       at java.net.URLClassLoader.findClass(URLClassLoader.java:602)
       at com.ibm.ws.bootstrap.ExtClassLoader.findClass(ExtClassLoader.java:243)
       at java.lang.ClassLoader.loadClassHelper(ClassLoader.java:777)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:754)
       at com.ibm.ws.bootstrap.ExtClassLoader.loadClass(ExtClassLoader.java:134)
       at java.lang.ClassLoader.loadClass(ClassLoader.java:731)

Given that this had worked before the change and given that we know what the change was, this was somewhat weird.

This is covered here: -

Configure IBM Business Process Manager to parse XMLType data in Oracle database through WebSphere JDBC Adapter

*BUT* nothing has changed … apart from what has changed.

We dug further through the logs, but to no avail.

We then started to dig into the WebSphere Classloader, using this as source: -

An Overview of the WebSphere Application Server Class Loader

and this: -

MustGather: Classloader problems for WebSphere Application Server

specifically by tracing the class loader : -

Enable Java™ Virtual Machine (JVM) classloader traces through the Admin console:

• Select Servers, choose Application servers and select the server you want to configure.

• In the Server Infrastructure section, open Java and Process Management and select Process Definition.

• Under Additional Properties, select Java Virtual Machine.

• Check the Verbose class loading checkbox.

• Add the following string to the Generic JVM arguments field:

-Dws.ext.debug=true  -Dws.osgi.debug

• Click OK.

Once we did this, and looked at the trace, we could see that, whilst the Oracle JDBC driver ( ojdbc7.jar ) *WAS* being loaded ( and we could see that, partly because BPM was coming up and working, and partly because the JDBC Test Connection function worked for all of the databases, both BPM and application ), the requisite Oracle XML tool ( xdb6.jar ) was NOT being loaded.

However, when we looked at the configuration of the JDBC driver that was being used by the application-specific data source, it looked to be in order, similar to this: -

( although we're using the ojdbc7.jar rather than ojdbc6.jar as per the screenshot above ).

When we checked the underlying Linux file-system, we could see that ojdbc7.jar was present : -

ls -al /opt/ibm/WebSphere/AppServer/jdbcdrivers/Oracle/

total 6632
drwxr-xr-x 2 wasadmin wasadmins      40 Jul 10 19:59 .
drwxr-xr-x 5 wasadmin wasadmins      45 Jul 10 19:59 ..
-rw-r--r-- 1 wasadmin wasadmins 3389454 Jul 10 19:59 ojdbc6.jar
-rw-r--r-- 1 wasadmin wasadmins 3397734 Jul 10 19:59 ojdbc7.jar

*BUT* that the xdb6.jar was NOT in the same location.

When we dug further, we could see that xdb6.jar was here: -

/opt/ibm/WebSphere/AppServer/lib/ext

along with ojdbc7.jar and xmlparserv2.jar.

which is interesting.

This then led the client to drill into the WebSphere Variable ORACLE_JDBC_DRIVER_PATH which is referenced at the JDBC driver level.

This was set to: -

${WAS_INSTALL_ROOT}/jdbcdrivers/Oracle

which is, in effect, this path: -

/opt/ibm/WebSphere/AppServer/jdbcdrivers/Oracle/

Looking at another environment, it was clear that, at some point in the past, this variable had been altered to point here: -

/opt/ibm/WebSphere/AppServer/lib/ext

rather than here; -

/opt/ibm/WebSphere/AppServer/jdbcdrivers/Oracle/

But, and here's the interesting part, the BPMConfig.sh script must've reset the variable back to the IBM-supplied default.

So, the moral of the story is two-fold; try and avoid altering IBM-provided variables AND learn how to debug the class loader in WAS :-)