Sunday, 30 November 2014

Oops, DB2 and the Insufficient Disk Space

So I was trying to create a second Deployment Environment to host a Process Server for IBM BPM Advanced 8.5.5, when I saw this: -

CWMCB0313E: An error occurred while attempting to connect to database PDWDB2: DB2 SQL Error: SQLCODE=-1762, SQLSTATE=08004, SQLERRMC=null, DRIVER=4.11.69. Error code = -1,762, SQLState = 08004, URL = null.

from the DE Wizard.

I immediately went to check DB2: -

db2 connect to pdwdb2

and saw this: -

SQL1762N  Unable to connect to database because there is not enough space to allocate active log files.  SQLSTATE=08004

When I checked the disk: -

df -kmh

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhel65-lv_root
                       18G   16G  453M  98% /
tmpfs                 931M   40K  931M   1% /dev/shm
/dev/sda1             477M   62M  391M  14% /boot
.host:/               1.1T  135G  904G  13% /mnt/hgfs

which isn't a good look.

Having allocated more space using VMware, it was a quick job to extend the file-system: -

fdisk /dev/sda
reboot 
pvcreate /dev/sda3
vgextend vg_rhel65 /dev/sda3
lvextend /dev/mapper/vg_rhel65-lv_root /dev/sda3 
resize2fs -p /dev/mapper/vg_rhel65-lv_root 

and now we're peachy: -

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_rhel65-lv_root
                       37G   16G   20G  46% /
tmpfs                 931M     0  931M   0% /dev/shm
/dev/sda1             477M   62M  391M  14% /boot
.host:/               1.1T  133G  906G  13% /mnt/hgfs


IBM BPM to DB2 - You WILL respect my authority

Can you say "Doh" ? I bet you can.

I'm building a multi-Deployment Environment IBM BPM Advanced 8.5.5 environment today, with one DE hosting Process Center and another DE hosting Process Server. Why, because I can / want to / need to :-)

So I hit this problem AGAIN: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/bootstrapProcessServerData.sh -clusterName AppCluster

Bootstraping data into cluster AppCluster

WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
java.lang.Exception: java.lang.reflect.InvocationTargetException
at com.ibm.bpm.bootstrap.BootstrapProcessServerDataHelper.loadInstallationGuid(BootstrapProcessServerDataHelper.java:534)

Caused by: java.lang.IllegalStateException: Failed to initialize registry
at com.lombardisoftware.core.Registry.getInstance(Registry.java:116)
at com.lombardisoftware.utility.spring.ProgrammaticTransactionSupport.<clinit>(ProgrammaticTransactionSupport.java:119)

Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'handlersMap': Cannot create inner bean 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#8d9b870e' of type [com.lombardisoftware.server.ejb.persistence.PSDefaultHandler] while setting bean property 'sourceMap' with key [TypedStringValue: value [Task], target type [null]]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.lombardisoftware.server.ejb.persistence.PSDefaultHandler#8d9b870e' defined in class path resource [registry.persistence.xml]: Cannot resolve reference to bean 'dao.task' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dao.task' defined in class path resource [registry.persistence.xml]: Instantiation of bean failed; nested exception is java.lang.ExceptionInInitializerError

Yes, that cost me (cough) an hour or so of faffing about, before i re-visited THIS post on MY blog: -


In essence, I was connected from BPM to DB2 using db2user1 which did NOT have adequate permissions to insert/update data inside the tables :-(

db2 grant dbadm on database to user db2user1

at which point everything started working.

To make it even worse, I then hit something very similar with the Support Cluster (SupCluster) which threw these exceptions up: -

SystemOut.log

[29/11/14 09:06:56:427 GMT] 00000001 WsServerImpl  E   WSVR0009E: Error occurred during startup
com.ibm.ws.exception.RuntimeError: Failed to query the BPM version from database [jdbc/PerformanceDB]. Please check the ffdc log for detail information.
For fresh installation scenario, please run the database initialization scripts under the corresponding database schema first; for upgrade or migration scenario, please upgrade your database to match with current product version first.
        at com.ibm.bpm.migration.database.ValidateDatabaseVersion.verifyStandardDB(ValidateDatabaseVersion.java:392)
        at com.ibm.bpm.migration.database.ValidateDatabaseVersion.verifyStandardDB4ND(ValidateDatabaseVersion.java:260)


FFDC Logs

[29/11/14 09:06:56:414 GMT]     FFDC Exception:com.ibm.db2.jcc.am.SqlSyntaxErrorException SourceId:com.ibm.bpm.migration.database.ValidateDatabaseVersion.verifyStandardDB ProbeId:102 Reporter:java.lang.String@7eb4dcd0
com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-551, SQLSTATE=42501, SQLERRMC=DB2USER1;SELECT;DB2USER1.LSW_SYSTEM_SCHEMA, DRIVER=4.11.69

The first set of errors looked the most severe; the second set immediately nailed it - yes, it was the SAME DARN ANSWER :-(

db2 grant dbadm on database to user db2user1

Yes, again, once I changed the grants, everything just magically started working :-)

Saturday, 29 November 2014

How to configure Common Event Infrastructure and cbebrowser for IBM Business Process Manager Advanced V8.5

Found this via Twitter and developerWorks Answers 


This blog is an effort to explain the steps required to configure Common Event Infrastructure (CEI) and cbebrowser for IBM Business Process Manager (BPM) Advanced V8.5 or later. The Common Event Infrastructure is not enabled by default in IBM Business Process Manager Advanced V8.5.5 and you must, therefore, set up CEI in the IBM BPM cell if you are planning to use it. You can follow the steps below to run the wsadmin commands to enable CEI and to install cbebrowser:

The steps are somewhat different to those which I've performed before - specifically, I didn't realise that one could run AdminTask.configEventServiceDB2DB to generate the required SQL scripts.

Also, I've not used the CBE Browser application before.

Something to try when I return to site next week .....

Friday, 28 November 2014

IBM SoftLayer, IBM Installation Manager, Red Hat Linux and fun

One of my colleagues asked for some help with WAS 8.5.0 on a Red Hat Enterprise Linux 6.6 VM .... running on IBM SoftLayer.

This is my first time on the SoftLayer cloud, so it's been a blast.

One thing that caught me out was the requirement for the SoftLayer VPN client. Whilst my colleague kindly linked me to an ArrayNetworks VPN client ( ArrayNetworksL3VPN_MAC ), this didn't seem to want to work ( connect ) from my Mac ( OS X 10.10.1 Yosemite ).

Thankfully, the SoftLayer website came to my aid: -


This quickly talked me through the process to set up a VPN connection using the Mac's native PPTP client.

Once I set this up, I was in like Flynn, with a SSH tunnel going to the RHEL VM.

This is what was installed, Linux-wise: -

uname -a

Linux <<hostname>> 2.6.32-504.1.3.el6.x86_64 #1 SMP Fri Oct 31 11:37:10 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/redhat-release

Red Hat Enterprise Linux Server release 6.6 (Santiago)

My colleague was trying to use a 32-bit version of IBM Installation Manager 1.5.

This was a bad move :-(

As soon as we tried to launch IIM: -

./tools/imcl -version

Imcl:
An error has occurred.  See the log file
/root/configuration/1417196820270.log


with this in the log: -

!SESSION Fri Nov 28 11:47:00 CST 2014 ------------------------------------------
!ENTRY org.eclipse.equinox.launcher 4 0 2014-11-28 11:47:00.281
!MESSAGE Exception launching the Eclipse Platform:
!STACK
java.lang.ClassNotFoundException: org.eclipse.core.runtime.adaptor.EclipseStarter
        at java.net.URLClassLoader.findClass(URLClassLoader.java:434)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:653)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:619)
        at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:617)
        at org.eclipse.equinox.launcher.Main.basicRun(Main.java:575)
        at org.eclipse.equinox.launcher.Main.run(Main.java:1408)
        at org.eclipse.equinox.launcher.Main.main(Main.java:1384)

I'm guessing that this was caused by (a) the version of IIM being used - 1.5 - or (b) the fact that it was 32-bit on a 64-bit OS or (c) the version of RHEL.

I downloaded IIM 1.8 ( 64-bit ), which worked OK :-)

./imcl -version

Installation Manager (install kit)
Version: 1.8.0
Internal Version: 1.8.0.20140902_1503
Architecture: 64-bit


Installation Manager (installed)
Version: 1.6.0
Internal Version: 1.6.0.20120831_1216
Architecture: 64-bit


Additionally, this also highlighted that IIM 1.6 was already installed.

Once I knew this, I was on a roll.

I then wanted to see what version of WAS my colleague had downloaded: -

/opt/IBM/InstallationManager/eclipse/tools/imcl listAvailablePackages -repositories /root/WAS-installs/ -features

com.ibm.websphere.BASE.v85_8.5.0.20120501_1108 : core.feature,ejbdeploy,thinclient,embeddablecontainer,samples,liberty,com.ibm.sdk.6_32bit,com.ibm.sdk.6_64bit

/opt/IBM/InstallationManager/eclipse/tools/imcl listAvailablePackages -repositories /root/WAS-installs/ -features -long

/root/WAS-installs/ : com.ibm.websphere.BASE.v85_8.5.0.20120501_1108 : IBM WebSphere Application Server : 8.5.0.0 : core.feature,ejbdeploy,thinclient,embeddablecontainer,samples,liberty,com.ibm.sdk.6_32bit,com.ibm.sdk.6_64bit

I knocked up a IIM response file: -

installWAS8502.rsp
 
<?xml version="1.0" encoding="UTF-8"?>
<!--The "acceptLicense" attribute has been deprecated. Use "-acceptLicense" command line option to accept license agreements.-->
<agent-input acceptLicense='true'>
<server>
<repository location='/root/WAS-installs/'/>
</server>
<profile id='IBM WebSphere Application Server V8.5' installLocation='/opt/IBM/WebSphere/AppServer'>
<data key='eclipseLocation' value='/opt/IBM/WebSphere/AppServer'/>
<data key='user.import.profile' value='false'/>
<data key='cic.selector.os' value='linux'/>
<data key='cic.selector.arch' value='x86'/>
<data key='cic.selector.ws' value='gtk'/>
<data key='cic.selector.nl' value='en'/>
</profile>
<install modify='false'>
<offering id='com.ibm.websphere.BASE.v85' version='8.5.0.20120501_1108' profile='IBM WebSphere Application Server V8.5' features='core.feature,ejbdeploy,thinclient,embeddablecontainer,samples,liberty,com.ibm.sdk.6_64bit' installFixes='none'/>
</install>
<preference name='com.ibm.cic.common.core.preferences.eclipseCache' value='/opt/IBM/IMShared'/>
<preference name='com.ibm.cic.common.core.preferences.connectTimeout' value='30'/>
<preference name='com.ibm.cic.common.core.preferences.readTimeout' value='45'/>
<preference name='com.ibm.cic.common.core.preferences.downloadAutoRetryCount' value='0'/>
<preference name='offering.service.repositories.areUsed' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.ssl.nonsecureMode' value='false'/>
<preference name='com.ibm.cic.common.core.preferences.http.disablePreemptiveAuthentication' value='false'/>
<preference name='http.ntlm.auth.kind' value='NTLM'/>
<preference name='http.ntlm.auth.enableIntegrated.win32' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.preserveDownloadedArtifacts' value='true'/>
<preference name='com.ibm.cic.common.core.preferences.keepFetchedFiles' value='false'/>
<preference name='PassportAdvantageIsEnabled' value='false'/>
<preference name='com.ibm.cic.common.core.preferences.searchForUpdates' value='false'/>
<preference name='com.ibm.cic.agent.ui.displayInternalVersion' value='false'/>
<preference name='com.ibm.cic.common.sharedUI.showErrorLog' value='true'/>
<preference name='com.ibm.cic.common.sharedUI.showWarningLog' value='true'/>
<preference name='com.ibm.cic.common.sharedUI.showNoteLog' value='true'/>
</agent-input>

and installed WAS: -

/opt/IBM/InstallationManager/eclipse/tools/imcl -input /root/installWAS8502.rsp -acceptLicense

Installed com.ibm.websphere.BASE.v85_8.5.0.20120501_1108 to the /opt/IBM/WebSphere/AppServer directory.

and then reported upon what's installed: -

/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

com.ibm.cic.agent_1.6.0.20120831_1216
com.ibm.websphere.BASE.v85_8.5.0.20120501_1108


/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledFeatures com.ibm.websphere.BASE.v85_8.5.0.20120501_1108

com.ibm.sdk.6_64bit
ejbdeploy
embeddablecontainer
liberty
samples
thinclient


/opt/IBM/WebSphere/AppServer/bin/versionInfo.sh
 
WVER0010I: Copyright (c) IBM Corporation 2002, 2012; All rights reserved.
WVER0012I: VersionInfo reporter version 1.15.1.48, dated 2/8/12

28-Nov-2014 10:49:18 java.util.prefs.FileSystemPreferences$2 run
INFO: Created user preferences directory.
--------------------------------------------------------------------------------
IBM WebSphere Product Installation Status Report
--------------------------------------------------------------------------------

Report at date and time 28 November 2014 10:49:18 CST

Installation
--------------------------------------------------------------------------------
Product Directory        /opt/IBM/WebSphere/AppServer
Version Directory        /opt/IBM/WebSphere/AppServer/properties/version
DTD Directory            /opt/IBM/WebSphere/AppServer/properties/version/dtd
Log Directory            /var/ibm/InstallationManager/logs

Product List
--------------------------------------------------------------------------------
BASE                     installed

Installed Product
--------------------------------------------------------------------------------
Name                  IBM WebSphere Application Server
Version               8.5.0.0
ID                    BASE
Build Level           gm1218.01
Build Date            5/1/12
Package               com.ibm.websphere.BASE.v85_8.5.0.20120501_1108
Architecture          x86-64 (64 bit)
Installed Features    IBM 64-bit WebSphere SDK for Java
                      WebSphere Application Server Full Profile
                      EJBDeploy tool for pre-EJB 3.0 modules
                      Embeddable EJB container
                      WebSphere Application Server Liberty Profile
                      Sample applications
                      Stand-alone thin clients and resource adapters

--------------------------------------------------------------------------------
End Installation Status Report
-----------------------------------------------------

and then created a base WAS profile: -

/opt/IBM/WebSphere/AppServer/bin/manageprofiles.sh -response wasProfile.rsp 

INSTCONFSUCCESS: Success: Profile AppSrv01 now exists. Please consult /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/AboutThisProfile.txt for more information about this profile.

and then started the server: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/startServer.sh wasServer 

ADMU0116I: Tool information is being logged in file
           /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/wasServer/startServer.log
ADMU0128I: Starting tool with the AppSrv01 profile
ADMU3100I: Reading configuration for server: wasServer
ADMU3200I: Server launched. Waiting for initialization status.
ADMU3000I: Server wasServer open for e-business; process id is 27139

and finished by logging into the admin. console: -


Now it's weekend time :-)

IBM BPM - Deleting a Deployment Environment

Not sure why this happens but, when I'm trying to delete an IBM BPM 8.5.5 Deployment Environment: -

/opt/IBM/WebSphere/AppServer/bin/BPMConfig.sh -delete -de ProcessServer -profile Dmgr01

I get this: -

Logging to file /opt/IBM/WebSphere/AppServer/logs/config/BPMConfig_20141128-063539.log.

Do you want to delete deployment environment?
1. yes
2. no
Input option number: 1
Before executing command to delete a deployment environment, it must at least stop the deployment environment.
Checking the state of deployment environment ProcessServer...
The deployment manager is started.
The node AppSrv01Node is started.
The node AppSrv02Node is started.
The deployment environment ProcessServer is stopped.
Deleting deployment environment ProcessServer...
CWMCB1151E: The application(s) HTM_PredefinedTasks_V8000_AppCluster, HTM_PredefinedTaskMsg_V8000_AppCluster must be uninstalled manually before deleting the deployment environment ProcessServer.
The 'BPMConfig.sh -delete -profile Dmgr01 -de ProcessServer' command failed. For more information, see the log file /opt/IBM/WebSphere/AppServer/logs/config/BPMConfig_20141128-063539.log.


So I deleted the two offending applications, and I was then able to delete the DE: -

/opt/IBM/WebSphere/AppServer/bin/BPMConfig.sh -delete -de ProcessServer -profile Dmgr01

Logging to file /opt/IBM/WebSphere/AppServer/logs/config/BPMConfig_20141128-063934.log.

Do you want to delete deployment environment?
1. yes
2. no
Input option number: 1
Before executing command to delete a deployment environment, it must at least stop the deployment environment.
Checking the state of deployment environment ProcessServer...
The deployment manager is started.
The node AppSrv01Node is started.
The node AppSrv02Node is started.
The deployment environment ProcessServer is stopped.
Deleting deployment environment ProcessServer...
The action of deleting the deployment environment does not delete the authentication aliases, or the users used by ProcessServer, or the folder '/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/dbscripts/ProcessServer'.
The 'BPMConfig.sh -delete -profile Dmgr01 -de ProcessServer' command completed successfully.


For the record, I could've also chosen to add the parameter: -

-acceptDeletionPrompt

to my command to avoid the Do you want to delete deployment environment? prompt.

Thursday, 27 November 2014

IBM Business Monitor - some FAQs

Some great stuff here on dwAnswers





IBM Integration Bus - Port Reporting

Following on from this earlier post: -


I'm working with a colleague to implement the same configuration in another environment.

Again, we're using IBM HTTP Server 8.5.5.2 in front of IBM Integration Bus 9.0.0.2, in order to allow requests ( SOAP over HTTP in this particular case ) to be routed to one of a number of Integration Nodes ( aka Brokers in the old tongue ).

This is partly to improve performance ( more engines to service requests ) and partly for resilience ( service can be maintained in the event of the loss of one Integration Node ).

So I have learned three new things today ( what's the bon mot about impossible things and breakfast ? Well, mine is learn new stuff before AND after breakfast !! ).

Firstly, one can report on the IIB configuration to see what ports an Integration Node is listening for HTTP connections, as follows: -

mqsireportproperties <<NAME OF INTEGRATION NODE>> -b httplistener -o HTTPConnector -a

which returns something like this: -

 HTTPConnector
  uuid='HTTPConnector'
  address=''
  port='7122'
  maxPostSize=''
  maxSavePostSize=''
  acceptCount=''
  compressableMimeTypes=''
  compression=''
  connectionLinger=''
  connectionTimeout=''
  maxHttpHeaderSize=''
  maxKeepAliveRequests=''
  maxThreads=''
  minSpareThreads=''
  noCompressionUserAgents=''
  restrictedUserAgents=''
  socketBuffer=''
  tcpNoDelay=''
  enableLookups='false'

It's worth noting that the port ( 7122 in the above example ) isn't the port on which the Integration Server ( formerly Execution Group) listens ( see below ).

Secondly, in order for IHS to route HTTP requests to IIB, there needs to be another, different, port configured, specifically at the Integration Server ( formerly Execution Group)  level, as evidenced by a modified mqsireportproperties command: - 

mqsireportproperties <<NAME OF INTEGRATION NODE>> -e default -o HTTPConnector -a

HTTPConnector
  uuid='HTTPConnector'
  userTraceLevel='none'
  traceLevel='none'
  userTraceFilter='none'
  traceFilter='none'
  port='8220'
  address=''
  maxPostSize=''
  maxSavePostSize=''
  acceptCount=''
  compressableMimeTypes=''
  compression=''
 connectionLinger=''
  connectionTimeout=''
  maxHttpHeaderSize=''
  maxKeepAliveRequests=''
  maxThreads=''
  minSpareThreads=''
  noCompressionUserAgents=''
  restrictedUserAgents=''
  socketBuffer=''
  tcpNoDelay=''
  explicitlySetPortNumber='8220'
  enableLookups=''
  enableMQListener=''
  shutdownDelay=''
  allowCrossConnectorPolling=''
  autoRespondHTTPHEADRequests=''
  integratedWindowsAuthentication=''
  iwaTimeoutSeconds='300'

In this example, port 8220 is explicitly set for this Integration Server.

That's the port that I needed to know about in order to configure IBM HTTP Server / WebSphere Plugin to route requests to that particular Integration Server ( one of many ).

Thirdly, and importantly, this port will NOT come up until the Integration Server is started.

I'd assumed that IIB was similar to WAS in that the TCP/IP ports come up as soon as the container ( IIB instead of a WAS JVM ) starts, but that is NOT the case.

Bottom line, one needs to start the Integration Server ( which hosts the flow(s) running on IIB, before the port will show up.

This surprised us, as we were busy using netstat and telnet but couldn't find out why the port wasn't listening :-(

Finally, and fourthly, the IBM Integration Bus Toolkit allows one to export a plugin configuration file suitable for the WebSphere Plugin.

Therefore, I could've simply asked my IIB admin guru to generate the file, rather than my having to hand-code it, or use Java.

However, that's the point of learning .........

Thanks to this IBM Technote for some guidance on mqsireportproperties: -

Using the mqsireportproperties command in IBM Integration Bus or WebSphere Message Broker for a single message flow

IBM HTTP Server - maintenance mode

One of my colleagues was looking for some intel. on this earlier this week.

The requirement is to have IBM HTTP Server (IHS) display a different banner page when WebSphere Application Server is placed into maintenance mode e.g. when the WAS cell is being shut down, recycled, upgraded etc.

After a few iterations, and a read of this: -


this is what I configured.

In essence, we place a single file in the IHS DocumentRoot: -

DocumentRoot "/opt/IBM/HTTPServer/htdocs"

called: -

/opt/IBM/HTTPServer/htdocs/serverdown.txt

*BUT* only when we want IHS to display the maintenance page.

IHS then uses the Apache mod_rewrite directive to check for this file and, only if, display a static HTML file: -

maintenance.html

<html>
<head><title>Server Down For Maintenance</title>
<body>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td>
<div align="center">
<span style="font-family:Times New Roman, Times, serif; font-weight:bold;color:#990000; font-size:30px;">
Site is currently down for maintenance.<br>Please try again later.
</span>
</div>
</td>
</tr>
</table>
</body>
</html>


Now I have IHS listening for HTTP connections on port 8080 and listening for HTTPS connections on port 8443.

Therefore, I configured mod_rewrite to do a number of things: -

(1) Route any incoming requests on 8080 to 8443
(2) Route all requests to a specific page - I'm using IBM Business Process Manager here, so I want everyone to go to the Process Center URL
(3) Check for the maintenance-mode indicator file and, if present, present the maintenance page back to the user

This requires me to have TWO discrete blocks in httpd.conf related to mod_rewrite as follows: -

Push everything on port 8080 to the Process Center URL on port 8443

LoadModule rewrite_module /opt/IBM/HTTPServer/modules/mod_rewrite.so
<ifModule mod_rewrite.c>
        RewriteEngine on
        RewriteCond %{SERVER_PORT} !^80$
        RewriteRule ^(.*)$ https://%{SERVER_NAME}:8443/ProcessCenter/ [R=301,L]
        RewriteLog logs/rewrite.log
        RewriteLogLevel 3
</ifModule>


Enable SSL, push everything to the Process Center URL, check for the maintenance-mode indicator file

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 8443
<VirtualHost *:8443>
        SSLEnable
        RewriteEngine On
        RewriteCond %{REQUEST_URI} !^/ProcessCenter/
        RewriteRule ^(.*)$ https://%{SERVER_NAME}:8443/ProcessCenter/ [R=301,L]
        RewriteCond %{DOCUMENT_ROOT}/serverdown.txt -f
        RewriteRule ^(.*)$ /maintenance.html [PT]
        RewriteLog logs/ssl_rewrite.log
        RewriteLogLevel 3
</VirtualHost>
KeyFile /opt/IBM/HTTPServer/ssl/keystore.kdb
SSLDisable


I've highlighted the mod_rewrite directives above. Also, note that we're logging the rewrite activities in two places: -

HTTP > HTTPS = rewrite.log

e.g.

192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (2) init rewrite engine with requested uri /
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (3) applying pattern '^(.*)$' to uri '/'
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (2) rewrite '/' -> 'https://bpm855.uk.ibm.com:8443/ProcessCenter/'
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (2) explicitly forcing redirect with https://bpm855.uk.ibm.com:8443/ProcessCenter/
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (1) escaping https://bpm855.uk.ibm.com:8443/ProcessCenter/ for redirect
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (1) redirect to https://bpm855.uk.ibm.com:8443/ProcessCenter/ [REDIRECT/301]


HTTPS = ssl_rewrite.log

192.168.1.70 - - [26/Nov/2014:18:55:29 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6401a448/initial] (3) applying pattern '^(.*)$' to uri '/ProcessCenter/login.jsp'
192.168.1.70 - - [26/Nov/2014:18:55:29 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6401a448/initial] (3) applying pattern '^(.*)$' to uri '/ProcessCenter/login.jsp'
192.168.1.70 - - [26/Nov/2014:18:55:29 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6401a448/initial] (1) pass through /ProcessCenter/login.jsp


In order to be able to test this, I created a small Bash script to turn maintenance on and off: -

maint.sh
 
#!/bin/bash

if [ $1 = "on" ] ; then
touch /opt/IBM/HTTPServer/htdocs/serverdown.txt
else 
rm /opt/IBM/HTTPServer/htdocs/serverdown.txt
fi


When I turn maintenance mode on: -

./maint.sh on

and hit IHS on port 8080: -


this is what I see in rewrite.log: -

192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (2) init rewrite engine with requested uri /
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (3) applying pattern '^(.*)$' to uri '/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (2) rewrite '/' -> 'https://bpm855.uk.ibm.com:8443/ProcessCenter/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (2) explicitly forcing redirect with https://bpm855.uk.ibm.com:8443/ProcessCenter/
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (1) escaping https://bpm855.uk.ibm.com:8443/ProcessCenter/ for redirect
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (1) redirect to https://bpm855.uk.ibm.com:8443/ProcessCenter/ [REDIRECT/301]


and this is what I see in ssl_rewrite.log: -

192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (2) init rewrite engine with requested uri /ProcessCenter/
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (3) applying pattern '^(.*)$' to uri '/ProcessCenter/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (3) applying pattern '^(.*)$' to uri '/ProcessCenter/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (2) rewrite '/ProcessCenter/' -> '/maintenance.html'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (2) forcing '/maintenance.html' to get passed through to next API URI-to-filename handler
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (2) init rewrite engine with requested uri /favicon.ico
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (3) applying pattern '^(.*)$' to uri '/favicon.ico'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (2) rewrite '/favicon.ico' -> 'https://bpm855.uk.ibm.com:8443/ProcessCenter/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (2) explicitly forcing redirect with https://bpm855.uk.ibm.com:8443/ProcessCenter/
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (1) escaping https://bpm855.uk.ibm.com:8443/ProcessCenter/ for redirect
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (1) redirect to https://bpm855.uk.ibm.com:8443/ProcessCenter/ [REDIRECT/301]


Once I turn maintenance mode off: -

./maint.sh off

things return to normal :-)

Oh, what fun :-)


Saturday, 22 November 2014

Help - The Proxy Ate My Process Designer

This is a problem that I recently saw with a client, and was able to reproduce, and more importantly, fix on my own environment.

But first some background, one of the IBM BPM's major features is the Eclipse-based development, Process Designer. This interacts directly with Process Center, and provides a collaborative rich-client integrated development environment.

Unlike other development tools, Process Designer can NOT function with a constant connection to the Process Center run-time, and this connectivity is predominantly via an HTTP connection.

It's also worth emphasising that, almost uniquely, Process Center "wraps" around Microsoft Internet Explorer, which is an absolutely crucial point ( and also explains why PD isn't available on any platform other than Microsoft Windows ).

The issue described here relates to the fact that Process Designer was unable to connect to Process Center.

Here I was using IBM BPM Advanced 8.5.5, but I'd previously seen it at my client's site using 8.5.0.1.

The most obvious symptom is this dialogue box: -


even though it is absolutely possible to access, and log into, Process Center via a web browser.

I also validated the eclipse.ini configuration file, which contained: -

....
-Djava.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
-Dcom.ibm.bpm.processcenter.url=https://bpm855.uk.ibm.com:8443
-Djava.ext.dirs="./AppClient/java/jre/lib/ext;./AppClient/lib;./AppClient/plugins"
...

which ties up with the URL used from within the web browser.

Process Designer does have a nifty log file. For me, this was located here: -

C:\IBM\Process Designer\V8.5\workspace\.metadata\.log

The path may vary from case to case. It's also worth noting the precise directory ( .metadata ) and file ( .log ) names, as they include preceding period characters.

Anyway, the log showed this: -

!SESSION 2014-11-22 08:55:49.490 -----------------------------------------------
eclipse.buildId=unknown
java.fullversion=JRE 1.6.0 IBM J9 2.6 Windows Server 2008 R2 x86-32 20131230_180580 (JIT enabled, AOT enabled)
J9VM - R26_Java626_SR7_20131230_1725_B180580
JIT  - r11.b05_20131003_47443.02
GC   - R26_Java626_SR7_20131230_1725_B180580
J9CL - 20131230_180580
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en
Framework arguments:  -dir ltr
Command-line arguments:  -os win32 -ws win32 -arch x86 -consoleLog -dir ltr -clean

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:55:59.271
!MESSAGE [teamworks.ae.product.splash.WLESplashHandlerFactory retrieveSystemHTTPProxies] found system HTTPS proxy ---  host: 127.0.0.1 port: 80

!ENTRY teamworks.ae.product 1 0 2014-11-22 08:55:59.443
!MESSAGE [InteractiveSplashHandler] Starting Authoring Environment. Bundle: teamworks.ae.product Version: 8.5.5.0 BPMRepo Prefix: https://bpm855.uk.ibm.com:8443

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:07.758
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] I/O exception (java.net.ConnectException) caught when processing request: Connection refused: connect

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:07.758
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] Retrying request

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:08.787
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] I/O exception (java.net.ConnectException) caught when processing request: Connection refused: connect

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:08.787
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] Retrying request

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:09.817
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] I/O exception (java.net.ConnectException) caught when processing request: Connection refused: connect

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:09.817
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] Retrying request

!ENTRY teamworks.ae.product 4 0 2014-11-22 08:56:10.815
!MESSAGE [InteractiveSplashHandler] Attempt to connect to the Process Center at URL: https://bpm855.uk.ibm.com:8443/bpm/repo/bootstrap failed.  Verify that the server is running.
!STACK 0
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:412)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:271)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:258)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:376)
at java.net.Socket.connect(Socket.java:546)
at java.net.Socket.connect(Socket.java:495)
at java.net.Socket.<init>(Socket.java:392)
at java.net.Socket.<init>(Socket.java:263)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at teamworks.ae.product.splash.InteractiveSplashHandler.bootstrap(InteractiveSplashHandler.java:674)
at teamworks.ae.product.splash.InteractiveSplashHandler.authenticate(InteractiveSplashHandler.java:622)
at teamworks.ae.product.splash.InteractiveSplashHandler.handleButtonOKWidgetSelected(InteractiveSplashHandler.java:387)
at teamworks.ae.product.splash.InteractiveSplashHandler.access$100(InteractiveSplashHandler.java:68)
at teamworks.ae.product.splash.InteractiveSplashHandler$3.keyTraversed(InteractiveSplashHandler.java:377)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:247)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1053)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1077)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1062)
at org.eclipse.swt.widgets.Control.traverse(Control.java:3693)
at org.eclipse.swt.widgets.Control.translateTraversal(Control.java:3675)
at org.eclipse.swt.widgets.Display.translateTraversal(Display.java:4691)
at org.eclipse.swt.widgets.Display.filterMessage(Display.java:1267)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3653)
at teamworks.ae.product.splash.InteractiveSplashHandler.doEventLoop(InteractiveSplashHandler.java:329)
at teamworks.ae.product.splash.InteractiveSplashHandler.init(InteractiveSplashHandler.java:177)
at org.eclipse.ui.internal.Workbench$8.run(Workbench.java:781)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.ui.internal.Workbench.createSplashWrapper(Workbench.java:797)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2456)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2427)
at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:670)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:663)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at teamworks.ae.product.TWApplication.start(TWApplication.java:64)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:619)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:574)
at org.eclipse.equinox.launcher.Main.run(Main.java:1407)
at org.eclipse.equinox.launcher.Main.main(Main.java:1383)

and I have highlighted the most interesting exceptions.

Of these, this one: -

!MESSAGE [teamworks.ae.product.splash.WLESplashHandlerFactory retrieveSystemHTTPProxies] found system HTTPS proxy ---  host: 127.0.0.1 port: 80

is absolutely crucial, as it led me to my initial hypothesis.

Now I first saw this issue on-site in August, but it took a very long time to get into a position to prove it.

I believed that the issue was that, in accordance with the underlying Microsoft Windows OS and Microsoft Internet Explorer browser, Process Designer is trying to leverage the system's default configuration to use a HTTP proxy server.

Now proxy servers usually sit at the edge of a network, and are used to protect network users from threats from external servers. We've had proxy servers almost as long as we have had the internet, and many corporations use the HTTP proxy to ensure that their staff do not access prohibited content ( often including so-called personal sites such as social networks, blogs, discussion services etc. ).

Anyway, back to the problem in hand.

It was my belief that Process Designer was being directed to the HTTP proxy, by an OS default setting ( typically controlled by Microsoft Active Directory via a Group Policy Object ), and that the proxy was blocking access to Process Center.

Due to access to the proxy configuration being tightly controlled, by AD GPO, I had no way to prove this on-site, leastways not until quite recently.

Therefore, on my own BPM 8.5.5 environment, I configured Windows/IE to use a ( admittedly non-existent ) proxy server: -


using the same configuration as my client ( 127.0.0.0.1 port 80 ).

At my client, I'm guessing that there's actually some HTTP proxy software running locally ( perhaps part of an anti-virus solution ).

Regardless, I've told Windows to send all traffic to 127.0.0.1:80.....

Guess what, Process Designer fails exactly as expected, even though Process Center is most definitely up-and-running.

Once I disabled the proxy, PD > PC again worked.

Now whilst it's perfectly acceptable to have a HTTP proxy, as outlined above, it seems illogical to have intranet traffic ( PC > PD ) going via the proxy, partly for inconvenience ( it doesn't work ! ) and party for performance ( putting intranet and internet traffic through a proxy server may not be a great idea, performance-wise ).

Again, it may NOT be a proxy server per se, but a local client-side application.

However, it's also possible to tell Windows/IE to avoid the proxy for certain given hosts, so I added the hostname of Process Center into the relevant configuration field: -


This time around, Process Designer worked like a dream.

It took a while ( mainly in order to get the right level of access to modify the Windows/IE configuration at my client's site ), but my hypothesis proved true there as well, and we were again able to get PD > PC working.

On a related note, a few days ago, we hit almost the same issue with IBM Rational Team Concert, where the RTC client was also trying to go via the proxy. Again, the same resolution ( adding the RTC server to the Exceptions dialogue, as outlined above ) worked a treat.

Simple when one knows how .....

Finally, in case it helps, in BPM 8.5.0.1, it was possible to force Process Designer to only use HTTP/HTTPS, rather than HTTP/HTTPS *and* RMI via JMS. It appears that, with 8.5.5.0, this is now the default.

Either way, one can check whether this is configured or not, as follows: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

WASX7209I: Connected to process "dmgr" on node bpm85Node1 using SOAP connector;  The type of process is: DeploymentManager
WASX7031I: For help, enter: "print Help.help()"
wsadmin>b=AdminConfig.getid('/Cell:/ServerCluster:AppCluster/BPMClusterConfigExtension:/BPMProcessCenter:/')
wsadmin>print AdminConfig.showAttribute(b,'httpProtocolOnly')
true

This made no difference, either with 8.5.0.1 or 8.5.5.0, but, if needed, one can toggle the setting on: -

AdminConfig.modify(b,[['httpProtocolOnly','true']]) 

or off: -

AdminConfig.modify(b,[['httpProtocolOnly','false']]) 

as documented here: -


*UPDATE*

If one ever needs to know how to configure the Windows Registry to enable the Connections/Proxy tab in IE to be edited, here you go: -

Registry keys that you will need to tweak, Change value from 1 (Disabled) to 0 (Enabled)

HKLM\Software\Policies\Microsoft\Internet Explorer\Control Panel

“Connection Settings”
“ConnectionsTab”
“Proxy”


Lessons Learned - WebSphere MQ, Clustering and Message Driven Beans

Context

The requirement is to create a clustered WebSphere MQ infrastructure, and then send messages to an application, known as a Message Driven Bean, deployed onto WebSphere Application Server.

The next step will be to create a more sophisticated application that can send and receive messages to/from WebSphere MQ, most likely leveraging the JavaEE Service Component Architecture.

In this scenario, I will create a pair of WMQ Queue Managers, each on a separate OS ( Red Hat VM ), one representing the Enterprise Service Bus (ESB), which will hold a full repository, and one representing IBM Business Process Manager, hosting a partial repository.

For me, the key lesson learned is that one creates a cluster Queue local to where it's actually used. In other words, because I need to interact with the clustered Queue on the BPM box, the queue needs to be hosted by the local ( to BPM ) Queue Manager.

WMQ Cluster Setup

Set up full repository on ESB box

Create Queue Manager

crtmqm ESB101_QM

Start Queue Manager

strmqm ESB101_QM

Start MQSC Environment

runmqsc ESB101_QM

Add a Repository Definition

ALTER QMGR REPOS(ESB101)

Define Listener

DEFINE LISTENER(TCP.LISTENER) TRPTYPE(tcp) CONTROL(qmgr) PORT(1414)
START LISTENER(TCP.LISTENER)

Define CLUSSDR Channel

DEFINE CHANNEL(BPM855) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('BPM855.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define CLUSRCVR Channel

DEFINE CHANNEL(ESB101) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('ESB.UK.IBM.COM(1414)') CLUSTER(ESB101)

Setup partial repository on BPM box

Create Queue Manager

crtmqm BPM855_QM

Start Queue Manager

strmqm BPM855_QM

Start MQSC Environment

runmqsc BPM855_QM

Define Listener

DEFINE LISTENER(TCP.LISTENER) TRPTYPE(tcp) CONTROL(qmgr) PORT(1414)
START LISTENER(TCP.LISTENER)

Define CLUSSDR Channel

DEFINE CHANNEL(ESB101) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('ESB.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define CLUSRCVR Channel

DEFINE CHANNEL(BPM855) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('BPM855.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define Cluster Queue

DEFINE QLOCAL(ESB101_Q) CLUSTER(ESB101)

Change Authority for Queue and Queue Manager to allow the WAS user (wasadmin) to connect - BPM box

setmqaut -m BPM855_QM -t qmgr -p wasadmin +connect +inq +dsp 
setmqaut -m BPM855_QM -t q -n ESB101_Q -p wasadmin +inq +browse +put 
setmqaut -m BPM855_QM -t q -n ESB101_Q -p wasadmin +inq +browse +get 

WMQ Cluster Testing

Put messages onto the cluster Queue - ESB box

/opt/mqm/samp/bin/amqsput ESB101_Q ESB101_QM

Sample AMQSPUT0 start
target queue is ESB101_Q
Hello
World

Sample AMQSPUT0 end

Get messages from the cluster queue - BPM box

/opt/mqm/samp/bin/amqsget ESB101_Q BPM855_QM

Sample AMQSGET0 start
message <Hello>
message <World>

no more messages
Sample AMQSGET0 end

WAS Message Driven Bean Setup - BPM box

Start WSAdmin client

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

Update WAS MQ Provider to support local bindings ( need to add native path )

AdminTask.manageWMQ('"WebSphere MQ Resource Adapter(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|resources.xml#J2CResourceAdapter_1416556034607)"', '[-nativePath /opt/mqm/java/lib64/ -disableWMQ false ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Create J2C Authentication Alias for wasadmin user

AdminTask.setAdminActiveSecuritySettings('[-customProperties["com.ibm.websphere.security.JAASAuthData.removeNodeNameGlobal=true"]]') 
AdminTask.createAuthDataEntry('[-alias wasadmin -user wasadmin -password passw0rd -description "WebSphere MQ Connectivity User" ]') 
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Create Queue and Activation Specification ( scoped to footer server )

AdminTask.createWMQQueue('foobar(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|server.xml)', '[-name ESB101_Q -jndiName jms/ESB101_Q -queueName ESB101_Q -qmgr -description ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

AdminTask.createWMQActivationSpec('"WebSphere MQ JMS Provider(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|resources.xml#builtin_mqprovider)"', '[-name ESB101_AS -jndiName jms/ESB101_AS -description -destinationJndiName jms/ESB101_Q -destinationType javax.jms.Queue -messageSelector -qmgrName BPM855_QM -wmqTransportType BINDINGS -qmgrSvrconnChannel -qmgrHostname -authAlias wasadmin]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Install Message Driven Bean

AdminApp.install('/home/wasadmin/SampleMDBEJB.ear', '[ -nopreCompileJSPs -distributeApp -nouseMetaDataFromBinary -nodeployejb -appname SampleMDBEJBEAR -createMBeansForResources -noreloadEnabled -nodeployws -validateinstall warn -noprocessEmbeddedConfig -filepermission .*\.dll=755#.*\.so=755#.*\.a=755#.*\.sl=755 -noallowDispatchRemoteInclude -noallowServiceRemoteInclude -asyncRequestDispatchType DISABLED -nouseAutoLink -noenableClientModule -clientMode isolated -novalidateSchema -MapModulesToServers [[ SampleMDBEJB SampleMDBEJB.jar,META-INF/ejb-jar.xml WebSphere:cell=bpm85Cell1,node=AppSrv01Node,server=foobar ]] -BindJndiForEJBMessageBinding [[ SampleMDBEJB SampleMDB SampleMDBEJB.jar,META-INF/ejb-jar.xml "" jms/ESB101_AS jms/ESB101_Q wasadmin ]]]' ) 
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Restart Server

AdminControl.invoke('WebSphere:name=foobar,process=foobar,platform=proxy,node=AppSrv01Node,j2eeType=J2EEServer,version=8.5.5.2,type=Server,mbeanIdentifier=cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar/server.xml#Server_1416599363822,cell=bpm85Cell1,spec=1.0,processType=ManagedProcess', 'restart', '[]', '[]')

WMQ Cluster > MDB Testing

On ESB box

/opt/mqm/samp/bin/amqsput ESB101_Q ESB101_QM

Sample AMQSPUT0 start
target queue is ESB101_Q
Hello
WebSphere
From
WMQ

Sample AMQSPUT0 end

On BPM box

cat /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/foobar/SystemOut.log

[22/11/14 07:57:30:958 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => Hello
[22/11/14 07:57:36:271 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => WebSphere
[22/11/14 07:57:38:475 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => From
[22/11/14 07:57:40:350 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => WMQ

Notes and Lessons Learned - 1

The credentials specified in the J2C Authentication Alias are crucial. At one point, I mis-specified the password, and saw the following exceptions in the WAS SystemOut.log : -

eMDBEJBEAR application is bound to the jms/ESB101_AS activation specification.
[22/11/14 08:01:38:857 GMT] 00000081 SibMessage    W   [:] CWSJY0003W: MQJCA4023: Startup reconnection failed for ActivationSpec 'javax.jms.Queue:jms/ESB101_Q@BPM855_QM <1648233785>'. Exception details: '
                       Message : com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'BPM855_QM' with connection mode 'Bindings' and host name 'localhost(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.

     Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').

[22/11/14 08:01:38:864 GMT] 00000081 ActivationSpe E   J2CA0138E: The Message Endpoint activation failed for ActivationSpec jms/ESB101_AS (com.ibm.mq.connector.inbound.ActivationSpecImpl) and MDB application SampleMDBEJBEAR#SampleMDBEJB.jar#SampleMDB due to the following exception: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.

Caused by: com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'BPM855_QM' with connection mode 'Bindings' and host name 'localhost(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.

[22/11/14 08:01:38:883 GMT] 00000081 RAWrapperImpl E   J2CA0089E: The method activateEndpoint on ResourceAdapter JavaBean cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar/resources.xml#J2CResourceAdapter_1416556034607 failed with the following exception: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.

with this in the WMQ log - /var/mqm/qmgrs/BPM855_QM/errors/AMQERR01.LOG  - for the local ( to WAS ) Queue Manager: -

AMQ5534: User ID 'wasadmin' authentication failed

EXPLANATION:
The user ID and password supplied by 'java' could not be authenticated.

AMQ5542: The failed authentication check was caused by the queue manager
CONNAUTH CHCKLOCL(OPTIONAL) configuration.

EXPLANATION:
The user ID 'wasadmin' and its password were checked because the queue manager
connection authority (CONNAUTH) configuration refers to an authentication
information (AUTHINFO) object named 'SYSTEM.DEFAULT.AUTHINFO.IDPWOS' with
CHCKLOCL(OPTIONAL).

I previously wrote about this here: -


in the context of the IBM Integration Bus Toolkit connecting to WMQ.

Bottom line, credentials are important :-)

Notes and Lessons Learned - 2

The biggest mistake I made was to assume that it was necessary to create the cluster Queue ( ESB101_Q of type QLOCAL ) on the ESB box. I spent many a fine hour trying to work out why things didn't work.

The point is that a client application ( amqsget or the MDB ) cannot connect to/use a Queue that's on a Queue Manager that's NOT local to the client. I'd assumed that I could make a bindings connection ( client to Queue Manager directly, without creating a client connection / channels etc. ) to a Queue Manager that's NOT local to the client/application.

This is why I do this job, every day is a school day, and I love to learn ...... :-)