Saturday, 22 November 2014

Lessons Learned - WebSphere MQ, Clustering and Message Driven Beans

Context

The requirement is to create a clustered WebSphere MQ infrastructure, and then send messages to an application, known as a Message Driven Bean, deployed onto WebSphere Application Server.

The next step will be to create a more sophisticated application that can send and receive messages to/from WebSphere MQ, most likely leveraging the JavaEE Service Component Architecture.

In this scenario, I will create a pair of WMQ Queue Managers, each on a separate OS ( Red Hat VM ), one representing the Enterprise Service Bus (ESB), which will hold a full repository, and one representing IBM Business Process Manager, hosting a partial repository.

For me, the key lesson learned is that one creates a cluster Queue local to where it's actually used. In other words, because I need to interact with the clustered Queue on the BPM box, the queue needs to be hosted by the local ( to BPM ) Queue Manager.

WMQ Cluster Setup

Set up full repository on ESB box

Create Queue Manager

crtmqm ESB101_QM

Start Queue Manager

strmqm ESB101_QM

Start MQSC Environment

runmqsc ESB101_QM

Add a Repository Definition

ALTER QMGR REPOS(ESB101)

Define Listener

DEFINE LISTENER(TCP.LISTENER) TRPTYPE(tcp) CONTROL(qmgr) PORT(1414)
START LISTENER(TCP.LISTENER)

Define CLUSSDR Channel

DEFINE CHANNEL(BPM855) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('BPM855.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define CLUSRCVR Channel

DEFINE CHANNEL(ESB101) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('ESB.UK.IBM.COM(1414)') CLUSTER(ESB101)

Setup partial repository on BPM box

Create Queue Manager

crtmqm BPM855_QM

Start Queue Manager

strmqm BPM855_QM

Start MQSC Environment

runmqsc BPM855_QM

Define Listener

DEFINE LISTENER(TCP.LISTENER) TRPTYPE(tcp) CONTROL(qmgr) PORT(1414)
START LISTENER(TCP.LISTENER)

Define CLUSSDR Channel

DEFINE CHANNEL(ESB101) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('ESB.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define CLUSRCVR Channel

DEFINE CHANNEL(BPM855) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('BPM855.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define Cluster Queue

DEFINE QLOCAL(ESB101_Q) CLUSTER(ESB101)

Change Authority for Queue and Queue Manager to allow the WAS user (wasadmin) to connect - BPM box

setmqaut -m BPM855_QM -t qmgr -p wasadmin +connect +inq +dsp 
setmqaut -m BPM855_QM -t q -n ESB101_Q -p wasadmin +inq +browse +put 
setmqaut -m BPM855_QM -t q -n ESB101_Q -p wasadmin +inq +browse +get 

WMQ Cluster Testing

Put messages onto the cluster Queue - ESB box

/opt/mqm/samp/bin/amqsput ESB101_Q ESB101_QM

Sample AMQSPUT0 start
target queue is ESB101_Q
Hello
World

Sample AMQSPUT0 end

Get messages from the cluster queue - BPM box

/opt/mqm/samp/bin/amqsget ESB101_Q BPM855_QM

Sample AMQSGET0 start
message <Hello>
message <World>

no more messages
Sample AMQSGET0 end

WAS Message Driven Bean Setup - BPM box

Start WSAdmin client

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

Update WAS MQ Provider to support local bindings ( need to add native path )

AdminTask.manageWMQ('"WebSphere MQ Resource Adapter(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|resources.xml#J2CResourceAdapter_1416556034607)"', '[-nativePath /opt/mqm/java/lib64/ -disableWMQ false ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Create J2C Authentication Alias for wasadmin user

AdminTask.setAdminActiveSecuritySettings('[-customProperties["com.ibm.websphere.security.JAASAuthData.removeNodeNameGlobal=true"]]') 
AdminTask.createAuthDataEntry('[-alias wasadmin -user wasadmin -password passw0rd -description "WebSphere MQ Connectivity User" ]') 
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Create Queue and Activation Specification ( scoped to footer server )

AdminTask.createWMQQueue('foobar(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|server.xml)', '[-name ESB101_Q -jndiName jms/ESB101_Q -queueName ESB101_Q -qmgr -description ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

AdminTask.createWMQActivationSpec('"WebSphere MQ JMS Provider(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|resources.xml#builtin_mqprovider)"', '[-name ESB101_AS -jndiName jms/ESB101_AS -description -destinationJndiName jms/ESB101_Q -destinationType javax.jms.Queue -messageSelector -qmgrName BPM855_QM -wmqTransportType BINDINGS -qmgrSvrconnChannel -qmgrHostname -authAlias wasadmin]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Install Message Driven Bean

AdminApp.install('/home/wasadmin/SampleMDBEJB.ear', '[ -nopreCompileJSPs -distributeApp -nouseMetaDataFromBinary -nodeployejb -appname SampleMDBEJBEAR -createMBeansForResources -noreloadEnabled -nodeployws -validateinstall warn -noprocessEmbeddedConfig -filepermission .*\.dll=755#.*\.so=755#.*\.a=755#.*\.sl=755 -noallowDispatchRemoteInclude -noallowServiceRemoteInclude -asyncRequestDispatchType DISABLED -nouseAutoLink -noenableClientModule -clientMode isolated -novalidateSchema -MapModulesToServers [[ SampleMDBEJB SampleMDBEJB.jar,META-INF/ejb-jar.xml WebSphere:cell=bpm85Cell1,node=AppSrv01Node,server=foobar ]] -BindJndiForEJBMessageBinding [[ SampleMDBEJB SampleMDB SampleMDBEJB.jar,META-INF/ejb-jar.xml "" jms/ESB101_AS jms/ESB101_Q wasadmin ]]]' ) 
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Restart Server

AdminControl.invoke('WebSphere:name=foobar,process=foobar,platform=proxy,node=AppSrv01Node,j2eeType=J2EEServer,version=8.5.5.2,type=Server,mbeanIdentifier=cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar/server.xml#Server_1416599363822,cell=bpm85Cell1,spec=1.0,processType=ManagedProcess', 'restart', '[]', '[]')

WMQ Cluster > MDB Testing

On ESB box

/opt/mqm/samp/bin/amqsput ESB101_Q ESB101_QM

Sample AMQSPUT0 start
target queue is ESB101_Q
Hello
WebSphere
From
WMQ

Sample AMQSPUT0 end

On BPM box

cat /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/foobar/SystemOut.log

[22/11/14 07:57:30:958 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => Hello
[22/11/14 07:57:36:271 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => WebSphere
[22/11/14 07:57:38:475 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => From
[22/11/14 07:57:40:350 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => WMQ

Notes and Lessons Learned - 1

The credentials specified in the J2C Authentication Alias are crucial. At one point, I mis-specified the password, and saw the following exceptions in the WAS SystemOut.log : -

eMDBEJBEAR application is bound to the jms/ESB101_AS activation specification.
[22/11/14 08:01:38:857 GMT] 00000081 SibMessage    W   [:] CWSJY0003W: MQJCA4023: Startup reconnection failed for ActivationSpec 'javax.jms.Queue:jms/ESB101_Q@BPM855_QM <1648233785>'. Exception details: '
                       Message : com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'BPM855_QM' with connection mode 'Bindings' and host name 'localhost(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.

     Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').

[22/11/14 08:01:38:864 GMT] 00000081 ActivationSpe E   J2CA0138E: The Message Endpoint activation failed for ActivationSpec jms/ESB101_AS (com.ibm.mq.connector.inbound.ActivationSpecImpl) and MDB application SampleMDBEJBEAR#SampleMDBEJB.jar#SampleMDB due to the following exception: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.

Caused by: com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'BPM855_QM' with connection mode 'Bindings' and host name 'localhost(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.

[22/11/14 08:01:38:883 GMT] 00000081 RAWrapperImpl E   J2CA0089E: The method activateEndpoint on ResourceAdapter JavaBean cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar/resources.xml#J2CResourceAdapter_1416556034607 failed with the following exception: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.

with this in the WMQ log - /var/mqm/qmgrs/BPM855_QM/errors/AMQERR01.LOG  - for the local ( to WAS ) Queue Manager: -

AMQ5534: User ID 'wasadmin' authentication failed

EXPLANATION:
The user ID and password supplied by 'java' could not be authenticated.

AMQ5542: The failed authentication check was caused by the queue manager
CONNAUTH CHCKLOCL(OPTIONAL) configuration.

EXPLANATION:
The user ID 'wasadmin' and its password were checked because the queue manager
connection authority (CONNAUTH) configuration refers to an authentication
information (AUTHINFO) object named 'SYSTEM.DEFAULT.AUTHINFO.IDPWOS' with
CHCKLOCL(OPTIONAL).

I previously wrote about this here: -


in the context of the IBM Integration Bus Toolkit connecting to WMQ.

Bottom line, credentials are important :-)

Notes and Lessons Learned - 2

The biggest mistake I made was to assume that it was necessary to create the cluster Queue ( ESB101_Q of type QLOCAL ) on the ESB box. I spent many a fine hour trying to work out why things didn't work.

The point is that a client application ( amqsget or the MDB ) cannot connect to/use a Queue that's on a Queue Manager that's NOT local to the client. I'd assumed that I could make a bindings connection ( client to Queue Manager directly, without creating a client connection / channels etc. ) to a Queue Manager that's NOT local to the client/application.

This is why I do this job, every day is a school day, and I love to learn ...... :-)

No comments:

Note to self - use kubectl to query images in a pod or deployment

In both cases, we use JSON ... For a deployment, we can do this: - kubectl get deployment foobar --namespace snafu --output jsonpath="{...