IBM Business Process Manager Advanced - Fun with Integration and Monitoring

So we had two interesting issues yesterday, in the context of a Business Process Execution Language (BPEL) application, using the Service Component Architecture (SCA) Java APIs.

Summary

- Ensure that the WAS configuration matches the application e.g. if the application uses Connection Factory, don't configure Queue Connection Factory

- Ensure that JMS Connection Factory / Queue Connection Factory objects are correctly configured for WebSphere MQ Queue Manager *IF* using clustered Queues

- Ensure that JMS Queue objects are correctly configured for WebSphere MQ Queue *IF* using clustered Queues

- Ensure that the Common Event Infrastructure (CEI) framework is correctly configured if running BPEL applications in one cluster ( AppCluster ), with CEI running on another cluster ( SupCluster )

Detail

The application writes messages out to an integration solution ( IBM Integration Bus via a WebSphere MQ cluster ), and also emits events to IBM Business Monitor ( BAM ), via the Service Integration Bus (SIBus) infrastructure.

The application has been written using IBM Integration Designer, and leverages the underlying WebSphere Application Server (WAS) platform, upon which BPM Advanced is deployed.

So the first issue that we saw was that, despite evidence to the contrary, the application was coded to utilise the Connection Factory interface, which is OK. However, the WAS configuration was built on the assumption that the code was actually using the Queue Connection Factory subclass of Connection Factory.

Therefore, the code was looking for CF whereas WAS was configured for QCF.

This was evidenced by the Resource References configuration in WAS, seen via Applications > Application Types > WebSphere Enterprise Applications, which did NOT allow us to browse/select the Queue Connection Factory, even though the QCF could be seen within the Resources > JMS > Queue Connection Factories view.

Initially, we thought this was a WAS scope ( cell/cluster/node/server ) issue, but both the application AND the QCF were scoped at the same level - AppCluster.

However, once we created a Connection Factory in place of the QCF, with the same settings, it showed up in the Resource References view.

We then saw WMQ-related issues in the logs, including: -

JMSWMQ2008: Failed to open MQ queue 'AUDITQUEUE'.

Having confirmed that this queue DID exist in the WebSphere MQ, we scratched our heads for a while ....

However, the solution was forthcoming :-)

In essence, the Queue AUDITQUEUE is hosted by a WMQ Cluster, which is "owned" by a remote ( to BPM ) Queue Manager, running on our ESB server ( IBM Integration Bus ).

Therefore, this Queue is, effectively, unknown to the local ( to BPM ) Queue Manager.

This means that, whilst it is possible to write messages to this Queue from the BPM server using, for example, the amqsput command: -

amqsput auditqueue localQM

the Queue is not "owned" by the local Queue Manager.

Bottom line, this means that the Queue Connection Factory or, in our case, the Connection Factory does NOT need to be configured to "know" about the Queue, even though there is an option to enter BOTH the Queue Manager AND Queue names.

Therefore, we changed the newly created CF to point at the local Queue Manager but NOT the clustered Queue.

However, we also have a JMS Queue, which IS used by the BPEL application, seen via the Applications > SCA Modules view in the WAS console. The JMS Queue only needs to "know" about the WMQ Queue, NOT the Queue Manager, which is "handled" by the Connection Factory.

Summary

JMS Connection Factory / Queue Connection Factory > only requires name of local Queue Manager

JMS Queue > only requires name of clustered Queue

This overcame the JMSWMQ2008 exception.

We then moved the problem on - the application starts OK, but then throws a bunch of JNDI-related exceptions during use.

This proved to be because the application emits events intended to be consumed by IBM Business Monitor.

Examples include: -

CEMEM0003E: The specified emitter factory was not found in JNDI. javax.naming.NameNotFoundException: Context: PSCell1/clusters/AppCluster, name: com/ibm/events/configuration/emitter/Default: First component in name events/configuration/emitter/Default not found. 

Caused by: javax.xml.ws.WebServiceException: javax.xml.ws.WebServiceException: com.ibm.websphere.sca.ServiceRuntimeException: com.ibm.bpe.api.EngineStateObserverEventEr

ror: CWWBE0013E: An error occurred during the event handling of 'activityFailed' in the observer plug-in type 'com.ibm.bpe.engine.observer.CEMStateObserverPlugin'.: cau

sed by: com.ibm.bpe.api.EngineStateObserverEventError: CWWBE0013E: An error occurred during the event handling of 'activityFailed' in the observer plug-in type 'com.ibm

.bpe.engine.observer.CEMStateObserverPlugin'.

Caused by: java.lang.IllegalStateException: Cannot access CEI

We spent a long time trying to resolve this.

Having confirmed that BPM was already happily emitting events to BAM, via the Service Integration Bus ( in essence, one configures a remote Messaging Engine, hosted by BAM, within the BPM SIBus configuration ). This means that the BPM applications emit, running on the AppCluster, emit events, which are passed via the Common Event Infrastructure (CEI) framework, running on the SupCluster, to the SIBus, hosted by the MECluster.

We had previously verified that BPM > BAM connectivity was working, using a Business Process Modelling Notation (BPMN) application.

However, our BPEL application was failing with the JNDI messages referenced previously :-(

After discussions with others in the wider WebSphere team ( specifically two of the world's best BAM SMEs ), we determined that the process that BPEL uses to emit events from BPM to BAM is completely different to that used by BPMN, which is good to know :-)

Eventually, we stumbled on the solution, partly through dialogue and partly through online searching.

In essence, it is necessary to "tell" the AppCluster, hosting the BPEL applications, that CEI is hosted by the SupCluster.

Ironically, this is documented here: -

Configuring Common Event Infrastructure (deprecated)

<snip>

Optional: If you are migrating to IBM BPM V8.5 from an earlier version, or if CEI and Business Process Choreographer are deployed on different clusters, configure the CEI emitter factory:
DMGR_PROFILE/bin/wsadmin -lang jython -f WAS_home\util\migration\scripts\setCEIDestination.py -a applicationClusterName -s supportClusterName -no-sync

Where:

-a applicationClusterName
is the name of the application cluster
-s supportClusterName
is the name of the support cluster
–no-sync

is an optional parameter that specifies whether to synchronize the changes to all nodes. By default, no-sync is false, which means that the node synchronize command is called immediately after the change. If no-sync is true, the node synchronize command is not called automatically and you must synchronize the nodes later so that the changes take effect.

An example of configuring the CEI emitter factory follows:

DMGR_PROFILE/bin/wsadmin -lang jython -f P:\bpm8500\util\migration\scripts\setCEIDestination.py -a AppCluster -s SupportCluster 

</snip>

Note the word Optional - that does NOT help :-) Also, note the phrase If you are migrating to IBM BPM V8.5 from an earlier version. Again, that doesn't help :-)

The important phrase is: -

if CEI and Business Process Choreographer are deployed on different clusters, configure the CEI emitter factory:

That IS the important part.

We're using the so-called Gold Standard BPM topology, aka Application, Remote Messaging, and Remote Support.

Therefore, we DO have CEI and BPC on different clusters, respectively SupCluster and AppCluster.

Following the so-called optional step, we "told" AppTarget where SupCluster is: -

opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -host `hostname` -port 8879 -username wasadmin -password passw0rd -f /home/wasadmin/setCEIDestination.py -a AppCluster -s SupCluster

WASX7209I: Connected to process "dmgr" on node dmgr using SOAP connector;  The type of process is: DeploymentManager

WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[-a, AppCluster, -s, SupCluster]"
XXXX0000I: Updating configuration on server AppClusterMember1 on node AppSrv01Node.
XXXX0000I: Current value: com/ibm/events/configuration/emitter/Default
XXXX0000I: New value    : cell/clusters/SupCluster/com/ibm/events/configuration/emitter/Default
XXXX0000I: Configuration is being saved.
XXXX0000I: Synchronizing the configuration across all nodes.

---------------------------------------------------------------
AdminNodeManagement:        Synchronize the active nodes
Usage: AdminNodeManagement.syncActiveNodes()
Return: If the command is successfully invoked, a value of 1 is returned.
---------------------------------------------------------------

AppSrv01Node
XXXX0000I: Done.

Once we did this, and restarted the entire BPM Deployment Environment, we no longer saw the JNDI / CEI exceptions, and events emitted by our BPEL application started showing up in BAM.