Friday, 28 November 2014

IBM BPM - Deleting a Deployment Environment

Not sure why this happens but, when I'm trying to delete an IBM BPM 8.5.5 Deployment Environment: -

/opt/IBM/WebSphere/AppServer/bin/BPMConfig.sh -delete -de ProcessServer -profile Dmgr01

I get this: -

Logging to file /opt/IBM/WebSphere/AppServer/logs/config/BPMConfig_20141128-063539.log.

Do you want to delete deployment environment?
1. yes
2. no
Input option number: 1
Before executing command to delete a deployment environment, it must at least stop the deployment environment.
Checking the state of deployment environment ProcessServer...
The deployment manager is started.
The node AppSrv01Node is started.
The node AppSrv02Node is started.
The deployment environment ProcessServer is stopped.
Deleting deployment environment ProcessServer...
CWMCB1151E: The application(s) HTM_PredefinedTasks_V8000_AppCluster, HTM_PredefinedTaskMsg_V8000_AppCluster must be uninstalled manually before deleting the deployment environment ProcessServer.
The 'BPMConfig.sh -delete -profile Dmgr01 -de ProcessServer' command failed. For more information, see the log file /opt/IBM/WebSphere/AppServer/logs/config/BPMConfig_20141128-063539.log.


So I deleted the two offending applications, and I was then able to delete the DE: -

/opt/IBM/WebSphere/AppServer/bin/BPMConfig.sh -delete -de ProcessServer -profile Dmgr01

Logging to file /opt/IBM/WebSphere/AppServer/logs/config/BPMConfig_20141128-063934.log.

Do you want to delete deployment environment?
1. yes
2. no
Input option number: 1
Before executing command to delete a deployment environment, it must at least stop the deployment environment.
Checking the state of deployment environment ProcessServer...
The deployment manager is started.
The node AppSrv01Node is started.
The node AppSrv02Node is started.
The deployment environment ProcessServer is stopped.
Deleting deployment environment ProcessServer...
The action of deleting the deployment environment does not delete the authentication aliases, or the users used by ProcessServer, or the folder '/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/dbscripts/ProcessServer'.
The 'BPMConfig.sh -delete -profile Dmgr01 -de ProcessServer' command completed successfully.


For the record, I could've also chosen to add the parameter: -

-acceptDeletionPrompt

to my command to avoid the Do you want to delete deployment environment? prompt.

Thursday, 27 November 2014

IBM Business Monitor - some FAQs

Some great stuff here on dwAnswers





IBM Integration Bus - Port Reporting

Following on from this earlier post: -


I'm working with a colleague to implement the same configuration in another environment.

Again, we're using IBM HTTP Server 8.5.5.2 in front of IBM Integration Bus 9.0.0.2, in order to allow requests ( SOAP over HTTP in this particular case ) to be routed to one of a number of Integration Nodes ( aka Brokers in the old tongue ).

This is partly to improve performance ( more engines to service requests ) and partly for resilience ( service can be maintained in the event of the loss of one Integration Node ).

So I have learned three new things today ( what's the bon mot about impossible things and breakfast ? Well, mine is learn new stuff before AND after breakfast !! ).

Firstly, one can report on the IIB configuration to see what ports an Integration Node is listening for HTTP connections, as follows: -

mqsireportproperties <<NAME OF INTEGRATION NODE>> -b httplistener -o HTTPConnector -a

which returns something like this: -

 HTTPConnector
  uuid='HTTPConnector'
  address=''
  port='7122'
  maxPostSize=''
  maxSavePostSize=''
  acceptCount=''
  compressableMimeTypes=''
  compression=''
  connectionLinger=''
  connectionTimeout=''
  maxHttpHeaderSize=''
  maxKeepAliveRequests=''
  maxThreads=''
  minSpareThreads=''
  noCompressionUserAgents=''
  restrictedUserAgents=''
  socketBuffer=''
  tcpNoDelay=''
  enableLookups='false'

It's worth noting that the port ( 7122 in the above example ) isn't the port on which the Integration Server ( formerly Execution Group) listens ( see below ).

Secondly, in order for IHS to route HTTP requests to IIB, there needs to be another, different, port configured, specifically at the Integration Server ( formerly Execution Group)  level, as evidenced by a modified mqsireportproperties command: - 

mqsireportproperties <<NAME OF INTEGRATION NODE>> -e default -o HTTPConnector -a

HTTPConnector
  uuid='HTTPConnector'
  userTraceLevel='none'
  traceLevel='none'
  userTraceFilter='none'
  traceFilter='none'
  port='8220'
  address=''
  maxPostSize=''
  maxSavePostSize=''
  acceptCount=''
  compressableMimeTypes=''
  compression=''
 connectionLinger=''
  connectionTimeout=''
  maxHttpHeaderSize=''
  maxKeepAliveRequests=''
  maxThreads=''
  minSpareThreads=''
  noCompressionUserAgents=''
  restrictedUserAgents=''
  socketBuffer=''
  tcpNoDelay=''
  explicitlySetPortNumber='8220'
  enableLookups=''
  enableMQListener=''
  shutdownDelay=''
  allowCrossConnectorPolling=''
  autoRespondHTTPHEADRequests=''
  integratedWindowsAuthentication=''
  iwaTimeoutSeconds='300'

In this example, port 8220 is explicitly set for this Integration Server.

That's the port that I needed to know about in order to configure IBM HTTP Server / WebSphere Plugin to route requests to that particular Integration Server ( one of many ).

Thirdly, and importantly, this port will NOT come up until the Integration Server is started.

I'd assumed that IIB was similar to WAS in that the TCP/IP ports come up as soon as the container ( IIB instead of a WAS JVM ) starts, but that is NOT the case.

Bottom line, one needs to start the Integration Server ( which hosts the flow(s) running on IIB, before the port will show up.

This surprised us, as we were busy using netstat and telnet but couldn't find out why the port wasn't listening :-(

Finally, and fourthly, the IBM Integration Bus Toolkit allows one to export a plugin configuration file suitable for the WebSphere Plugin.

Therefore, I could've simply asked my IIB admin guru to generate the file, rather than my having to hand-code it, or use Java.

However, that's the point of learning .........

Thanks to this IBM Technote for some guidance on mqsireportproperties: -

Using the mqsireportproperties command in IBM Integration Bus or WebSphere Message Broker for a single message flow

IBM HTTP Server - maintenance mode

One of my colleagues was looking for some intel. on this earlier this week.

The requirement is to have IBM HTTP Server (IHS) display a different banner page when WebSphere Application Server is placed into maintenance mode e.g. when the WAS cell is being shut down, recycled, upgraded etc.

After a few iterations, and a read of this: -


this is what I configured.

In essence, we place a single file in the IHS DocumentRoot: -

DocumentRoot "/opt/IBM/HTTPServer/htdocs"

called: -

/opt/IBM/HTTPServer/htdocs/serverdown.txt

*BUT* only when we want IHS to display the maintenance page.

IHS then uses the Apache mod_rewrite directive to check for this file and, only if, display a static HTML file: -

maintenance.html

<html>
<head><title>Server Down For Maintenance</title>
<body>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr>
<td>
<div align="center">
<span style="font-family:Times New Roman, Times, serif; font-weight:bold;color:#990000; font-size:30px;">
Site is currently down for maintenance.<br>Please try again later.
</span>
</div>
</td>
</tr>
</table>
</body>
</html>


Now I have IHS listening for HTTP connections on port 8080 and listening for HTTPS connections on port 8443.

Therefore, I configured mod_rewrite to do a number of things: -

(1) Route any incoming requests on 8080 to 8443
(2) Route all requests to a specific page - I'm using IBM Business Process Manager here, so I want everyone to go to the Process Center URL
(3) Check for the maintenance-mode indicator file and, if present, present the maintenance page back to the user

This requires me to have TWO discrete blocks in httpd.conf related to mod_rewrite as follows: -

Push everything on port 8080 to the Process Center URL on port 8443

LoadModule rewrite_module /opt/IBM/HTTPServer/modules/mod_rewrite.so
<ifModule mod_rewrite.c>
        RewriteEngine on
        RewriteCond %{SERVER_PORT} !^80$
        RewriteRule ^(.*)$ https://%{SERVER_NAME}:8443/ProcessCenter/ [R=301,L]
        RewriteLog logs/rewrite.log
        RewriteLogLevel 3
</ifModule>


Enable SSL, push everything to the Process Center URL, check for the maintenance-mode indicator file

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 8443
<VirtualHost *:8443>
        SSLEnable
        RewriteEngine On
        RewriteCond %{REQUEST_URI} !^/ProcessCenter/
        RewriteRule ^(.*)$ https://%{SERVER_NAME}:8443/ProcessCenter/ [R=301,L]
        RewriteCond %{DOCUMENT_ROOT}/serverdown.txt -f
        RewriteRule ^(.*)$ /maintenance.html [PT]
        RewriteLog logs/ssl_rewrite.log
        RewriteLogLevel 3
</VirtualHost>
KeyFile /opt/IBM/HTTPServer/ssl/keystore.kdb
SSLDisable


I've highlighted the mod_rewrite directives above. Also, note that we're logging the rewrite activities in two places: -

HTTP > HTTPS = rewrite.log

e.g.

192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (2) init rewrite engine with requested uri /
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (3) applying pattern '^(.*)$' to uri '/'
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (2) rewrite '/' -> 'https://bpm855.uk.ibm.com:8443/ProcessCenter/'
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (2) explicitly forcing redirect with https://bpm855.uk.ibm.com:8443/ProcessCenter/
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (1) escaping https://bpm855.uk.ibm.com:8443/ProcessCenter/ for redirect
192.168.1.70 - - [25/Nov/2014:21:50:49 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c64002968/initial] (1) redirect to https://bpm855.uk.ibm.com:8443/ProcessCenter/ [REDIRECT/301]


HTTPS = ssl_rewrite.log

192.168.1.70 - - [26/Nov/2014:18:55:29 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6401a448/initial] (3) applying pattern '^(.*)$' to uri '/ProcessCenter/login.jsp'
192.168.1.70 - - [26/Nov/2014:18:55:29 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6401a448/initial] (3) applying pattern '^(.*)$' to uri '/ProcessCenter/login.jsp'
192.168.1.70 - - [26/Nov/2014:18:55:29 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6401a448/initial] (1) pass through /ProcessCenter/login.jsp


In order to be able to test this, I created a small Bash script to turn maintenance on and off: -

maint.sh
 
#!/bin/bash

if [ $1 = "on" ] ; then
touch /opt/IBM/HTTPServer/htdocs/serverdown.txt
else 
rm /opt/IBM/HTTPServer/htdocs/serverdown.txt
fi


When I turn maintenance mode on: -

./maint.sh on

and hit IHS on port 8080: -


this is what I see in rewrite.log: -

192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (2) init rewrite engine with requested uri /
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (3) applying pattern '^(.*)$' to uri '/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (2) rewrite '/' -> 'https://bpm855.uk.ibm.com:8443/ProcessCenter/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (2) explicitly forcing redirect with https://bpm855.uk.ibm.com:8443/ProcessCenter/
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (1) escaping https://bpm855.uk.ibm.com:8443/ProcessCenter/ for redirect
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#13c1930][rid#7f1c5c015f98/initial] (1) redirect to https://bpm855.uk.ibm.com:8443/ProcessCenter/ [REDIRECT/301]


and this is what I see in ssl_rewrite.log: -

192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (2) init rewrite engine with requested uri /ProcessCenter/
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (3) applying pattern '^(.*)$' to uri '/ProcessCenter/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (3) applying pattern '^(.*)$' to uri '/ProcessCenter/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (2) rewrite '/ProcessCenter/' -> '/maintenance.html'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c6405c208/initial] (2) forcing '/maintenance.html' to get passed through to next API URI-to-filename handler
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (2) init rewrite engine with requested uri /favicon.ico
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (3) applying pattern '^(.*)$' to uri '/favicon.ico'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (2) rewrite '/favicon.ico' -> 'https://bpm855.uk.ibm.com:8443/ProcessCenter/'
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (2) explicitly forcing redirect with https://bpm855.uk.ibm.com:8443/ProcessCenter/
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (1) escaping https://bpm855.uk.ibm.com:8443/ProcessCenter/ for redirect
192.168.1.70 - - [27/Nov/2014:07:11:23 +0000] [bpm855.uk.ibm.com/sid#14326c8][rid#7f1c64018438/initial] (1) redirect to https://bpm855.uk.ibm.com:8443/ProcessCenter/ [REDIRECT/301]


Once I turn maintenance mode off: -

./maint.sh off

things return to normal :-)

Oh, what fun :-)


Saturday, 22 November 2014

Help - The Proxy Ate My Process Designer

This is a problem that I recently saw with a client, and was able to reproduce, and more importantly, fix on my own environment.

But first some background, one of the IBM BPM's major features is the Eclipse-based development, Process Designer. This interacts directly with Process Center, and provides a collaborative rich-client integrated development environment.

Unlike other development tools, Process Designer can NOT function with a constant connection to the Process Center run-time, and this connectivity is predominantly via an HTTP connection.

It's also worth emphasising that, almost uniquely, Process Center "wraps" around Microsoft Internet Explorer, which is an absolutely crucial point ( and also explains why PD isn't available on any platform other than Microsoft Windows ).

The issue described here relates to the fact that Process Designer was unable to connect to Process Center.

Here I was using IBM BPM Advanced 8.5.5, but I'd previously seen it at my client's site using 8.5.0.1.

The most obvious symptom is this dialogue box: -


even though it is absolutely possible to access, and log into, Process Center via a web browser.

I also validated the eclipse.ini configuration file, which contained: -

....
-Djava.naming.factory.initial=com.ibm.websphere.naming.WsnInitialContextFactory
-Dcom.ibm.bpm.processcenter.url=https://bpm855.uk.ibm.com:8443
-Djava.ext.dirs="./AppClient/java/jre/lib/ext;./AppClient/lib;./AppClient/plugins"
...

which ties up with the URL used from within the web browser.

Process Designer does have a nifty log file. For me, this was located here: -

C:\IBM\Process Designer\V8.5\workspace\.metadata\.log

The path may vary from case to case. It's also worth noting the precise directory ( .metadata ) and file ( .log ) names, as they include preceding period characters.

Anyway, the log showed this: -

!SESSION 2014-11-22 08:55:49.490 -----------------------------------------------
eclipse.buildId=unknown
java.fullversion=JRE 1.6.0 IBM J9 2.6 Windows Server 2008 R2 x86-32 20131230_180580 (JIT enabled, AOT enabled)
J9VM - R26_Java626_SR7_20131230_1725_B180580
JIT  - r11.b05_20131003_47443.02
GC   - R26_Java626_SR7_20131230_1725_B180580
J9CL - 20131230_180580
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en
Framework arguments:  -dir ltr
Command-line arguments:  -os win32 -ws win32 -arch x86 -consoleLog -dir ltr -clean

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:55:59.271
!MESSAGE [teamworks.ae.product.splash.WLESplashHandlerFactory retrieveSystemHTTPProxies] found system HTTPS proxy ---  host: 127.0.0.1 port: 80

!ENTRY teamworks.ae.product 1 0 2014-11-22 08:55:59.443
!MESSAGE [InteractiveSplashHandler] Starting Authoring Environment. Bundle: teamworks.ae.product Version: 8.5.5.0 BPMRepo Prefix: https://bpm855.uk.ibm.com:8443

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:07.758
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] I/O exception (java.net.ConnectException) caught when processing request: Connection refused: connect

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:07.758
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] Retrying request

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:08.787
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] I/O exception (java.net.ConnectException) caught when processing request: Connection refused: connect

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:08.787
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] Retrying request

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:09.817
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] I/O exception (java.net.ConnectException) caught when processing request: Connection refused: connect

!ENTRY teamworks.appserver.websphere 1 0 2014-11-22 08:56:09.817
!MESSAGE [org.apache.commons.httpclient.HttpMethodDirector executeWithRetry] Retrying request

!ENTRY teamworks.ae.product 4 0 2014-11-22 08:56:10.815
!MESSAGE [InteractiveSplashHandler] Attempt to connect to the Process Center at URL: https://bpm855.uk.ibm.com:8443/bpm/repo/bootstrap failed.  Verify that the server is running.
!STACK 0
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:412)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:271)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:258)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:376)
at java.net.Socket.connect(Socket.java:546)
at java.net.Socket.connect(Socket.java:495)
at java.net.Socket.<init>(Socket.java:392)
at java.net.Socket.<init>(Socket.java:263)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
at teamworks.ae.product.splash.InteractiveSplashHandler.bootstrap(InteractiveSplashHandler.java:674)
at teamworks.ae.product.splash.InteractiveSplashHandler.authenticate(InteractiveSplashHandler.java:622)
at teamworks.ae.product.splash.InteractiveSplashHandler.handleButtonOKWidgetSelected(InteractiveSplashHandler.java:387)
at teamworks.ae.product.splash.InteractiveSplashHandler.access$100(InteractiveSplashHandler.java:68)
at teamworks.ae.product.splash.InteractiveSplashHandler$3.keyTraversed(InteractiveSplashHandler.java:377)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:247)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1053)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1077)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1062)
at org.eclipse.swt.widgets.Control.traverse(Control.java:3693)
at org.eclipse.swt.widgets.Control.translateTraversal(Control.java:3675)
at org.eclipse.swt.widgets.Display.translateTraversal(Display.java:4691)
at org.eclipse.swt.widgets.Display.filterMessage(Display.java:1267)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3653)
at teamworks.ae.product.splash.InteractiveSplashHandler.doEventLoop(InteractiveSplashHandler.java:329)
at teamworks.ae.product.splash.InteractiveSplashHandler.init(InteractiveSplashHandler.java:177)
at org.eclipse.ui.internal.Workbench$8.run(Workbench.java:781)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
at org.eclipse.ui.internal.Workbench.createSplashWrapper(Workbench.java:797)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2456)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:2427)
at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:670)
at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:663)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:149)
at teamworks.ae.product.TWApplication.start(TWApplication.java:64)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:619)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:574)
at org.eclipse.equinox.launcher.Main.run(Main.java:1407)
at org.eclipse.equinox.launcher.Main.main(Main.java:1383)

and I have highlighted the most interesting exceptions.

Of these, this one: -

!MESSAGE [teamworks.ae.product.splash.WLESplashHandlerFactory retrieveSystemHTTPProxies] found system HTTPS proxy ---  host: 127.0.0.1 port: 80

is absolutely crucial, as it led me to my initial hypothesis.

Now I first saw this issue on-site in August, but it took a very long time to get into a position to prove it.

I believed that the issue was that, in accordance with the underlying Microsoft Windows OS and Microsoft Internet Explorer browser, Process Designer is trying to leverage the system's default configuration to use a HTTP proxy server.

Now proxy servers usually sit at the edge of a network, and are used to protect network users from threats from external servers. We've had proxy servers almost as long as we have had the internet, and many corporations use the HTTP proxy to ensure that their staff do not access prohibited content ( often including so-called personal sites such as social networks, blogs, discussion services etc. ).

Anyway, back to the problem in hand.

It was my belief that Process Designer was being directed to the HTTP proxy, by an OS default setting ( typically controlled by Microsoft Active Directory via a Group Policy Object ), and that the proxy was blocking access to Process Center.

Due to access to the proxy configuration being tightly controlled, by AD GPO, I had no way to prove this on-site, leastways not until quite recently.

Therefore, on my own BPM 8.5.5 environment, I configured Windows/IE to use a ( admittedly non-existent ) proxy server: -


using the same configuration as my client ( 127.0.0.0.1 port 80 ).

At my client, I'm guessing that there's actually some HTTP proxy software running locally ( perhaps part of an anti-virus solution ).

Regardless, I've told Windows to send all traffic to 127.0.0.1:80.....

Guess what, Process Designer fails exactly as expected, even though Process Center is most definitely up-and-running.

Once I disabled the proxy, PD > PC again worked.

Now whilst it's perfectly acceptable to have a HTTP proxy, as outlined above, it seems illogical to have intranet traffic ( PC > PD ) going via the proxy, partly for inconvenience ( it doesn't work ! ) and party for performance ( putting intranet and internet traffic through a proxy server may not be a great idea, performance-wise ).

Again, it may NOT be a proxy server per se, but a local client-side application.

However, it's also possible to tell Windows/IE to avoid the proxy for certain given hosts, so I added the hostname of Process Center into the relevant configuration field: -


This time around, Process Designer worked like a dream.

It took a while ( mainly in order to get the right level of access to modify the Windows/IE configuration at my client's site ), but my hypothesis proved true there as well, and we were again able to get PD > PC working.

On a related note, a few days ago, we hit almost the same issue with IBM Rational Team Concert, where the RTC client was also trying to go via the proxy. Again, the same resolution ( adding the RTC server to the Exceptions dialogue, as outlined above ) worked a treat.

Simple when one knows how .....

Finally, in case it helps, in BPM 8.5.0.1, it was possible to force Process Designer to only use HTTP/HTTPS, rather than HTTP/HTTPS *and* RMI via JMS. It appears that, with 8.5.5.0, this is now the default.

Either way, one can check whether this is configured or not, as follows: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

WASX7209I: Connected to process "dmgr" on node bpm85Node1 using SOAP connector;  The type of process is: DeploymentManager
WASX7031I: For help, enter: "print Help.help()"
wsadmin>b=AdminConfig.getid('/Cell:/ServerCluster:AppCluster/BPMClusterConfigExtension:/BPMProcessCenter:/')
wsadmin>print AdminConfig.showAttribute(b,'httpProtocolOnly')
true

This made no difference, either with 8.5.0.1 or 8.5.5.0, but, if needed, one can toggle the setting on: -

AdminConfig.modify(b,[['httpProtocolOnly','true']]) 

or off: -

AdminConfig.modify(b,[['httpProtocolOnly','false']]) 

as documented here: -


*UPDATE*

If one ever needs to know how to configure the Windows Registry to enable the Connections/Proxy tab in IE to be edited, here you go: -

Registry keys that you will need to tweak, Change value from 1 (Disabled) to 0 (Enabled)

HKLM\Software\Policies\Microsoft\Internet Explorer\Control Panel

“Connection Settings”
“ConnectionsTab”
“Proxy”


Lessons Learned - WebSphere MQ, Clustering and Message Driven Beans

Context

The requirement is to create a clustered WebSphere MQ infrastructure, and then send messages to an application, known as a Message Driven Bean, deployed onto WebSphere Application Server.

The next step will be to create a more sophisticated application that can send and receive messages to/from WebSphere MQ, most likely leveraging the JavaEE Service Component Architecture.

In this scenario, I will create a pair of WMQ Queue Managers, each on a separate OS ( Red Hat VM ), one representing the Enterprise Service Bus (ESB), which will hold a full repository, and one representing IBM Business Process Manager, hosting a partial repository.

For me, the key lesson learned is that one creates a cluster Queue local to where it's actually used. In other words, because I need to interact with the clustered Queue on the BPM box, the queue needs to be hosted by the local ( to BPM ) Queue Manager.

WMQ Cluster Setup

Set up full repository on ESB box

Create Queue Manager

crtmqm ESB101_QM

Start Queue Manager

strmqm ESB101_QM

Start MQSC Environment

runmqsc ESB101_QM

Add a Repository Definition

ALTER QMGR REPOS(ESB101)

Define Listener

DEFINE LISTENER(TCP.LISTENER) TRPTYPE(tcp) CONTROL(qmgr) PORT(1414)
START LISTENER(TCP.LISTENER)

Define CLUSSDR Channel

DEFINE CHANNEL(BPM855) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('BPM855.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define CLUSRCVR Channel

DEFINE CHANNEL(ESB101) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('ESB.UK.IBM.COM(1414)') CLUSTER(ESB101)

Setup partial repository on BPM box

Create Queue Manager

crtmqm BPM855_QM

Start Queue Manager

strmqm BPM855_QM

Start MQSC Environment

runmqsc BPM855_QM

Define Listener

DEFINE LISTENER(TCP.LISTENER) TRPTYPE(tcp) CONTROL(qmgr) PORT(1414)
START LISTENER(TCP.LISTENER)

Define CLUSSDR Channel

DEFINE CHANNEL(ESB101) CHLTYPE(CLUSSDR) TRPTYPE(TCP) CONNAME('ESB.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define CLUSRCVR Channel

DEFINE CHANNEL(BPM855) CHLTYPE(CLUSRCVR) TRPTYPE(TCP) CONNAME('BPM855.UK.IBM.COM(1414)') CLUSTER(ESB101)

Define Cluster Queue

DEFINE QLOCAL(ESB101_Q) CLUSTER(ESB101)

Change Authority for Queue and Queue Manager to allow the WAS user (wasadmin) to connect - BPM box

setmqaut -m BPM855_QM -t qmgr -p wasadmin +connect +inq +dsp 
setmqaut -m BPM855_QM -t q -n ESB101_Q -p wasadmin +inq +browse +put 
setmqaut -m BPM855_QM -t q -n ESB101_Q -p wasadmin +inq +browse +get 

WMQ Cluster Testing

Put messages onto the cluster Queue - ESB box

/opt/mqm/samp/bin/amqsput ESB101_Q ESB101_QM

Sample AMQSPUT0 start
target queue is ESB101_Q
Hello
World

Sample AMQSPUT0 end

Get messages from the cluster queue - BPM box

/opt/mqm/samp/bin/amqsget ESB101_Q BPM855_QM

Sample AMQSGET0 start
message <Hello>
message <World>

no more messages
Sample AMQSGET0 end

WAS Message Driven Bean Setup - BPM box

Start WSAdmin client

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd

Update WAS MQ Provider to support local bindings ( need to add native path )

AdminTask.manageWMQ('"WebSphere MQ Resource Adapter(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|resources.xml#J2CResourceAdapter_1416556034607)"', '[-nativePath /opt/mqm/java/lib64/ -disableWMQ false ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Create J2C Authentication Alias for wasadmin user

AdminTask.setAdminActiveSecuritySettings('[-customProperties["com.ibm.websphere.security.JAASAuthData.removeNodeNameGlobal=true"]]') 
AdminTask.createAuthDataEntry('[-alias wasadmin -user wasadmin -password passw0rd -description "WebSphere MQ Connectivity User" ]') 
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Create Queue and Activation Specification ( scoped to footer server )

AdminTask.createWMQQueue('foobar(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|server.xml)', '[-name ESB101_Q -jndiName jms/ESB101_Q -queueName ESB101_Q -qmgr -description ]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

AdminTask.createWMQActivationSpec('"WebSphere MQ JMS Provider(cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar|resources.xml#builtin_mqprovider)"', '[-name ESB101_AS -jndiName jms/ESB101_AS -description -destinationJndiName jms/ESB101_Q -destinationType javax.jms.Queue -messageSelector -qmgrName BPM855_QM -wmqTransportType BINDINGS -qmgrSvrconnChannel -qmgrHostname -authAlias wasadmin]')
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Install Message Driven Bean

AdminApp.install('/home/wasadmin/SampleMDBEJB.ear', '[ -nopreCompileJSPs -distributeApp -nouseMetaDataFromBinary -nodeployejb -appname SampleMDBEJBEAR -createMBeansForResources -noreloadEnabled -nodeployws -validateinstall warn -noprocessEmbeddedConfig -filepermission .*\.dll=755#.*\.so=755#.*\.a=755#.*\.sl=755 -noallowDispatchRemoteInclude -noallowServiceRemoteInclude -asyncRequestDispatchType DISABLED -nouseAutoLink -noenableClientModule -clientMode isolated -novalidateSchema -MapModulesToServers [[ SampleMDBEJB SampleMDBEJB.jar,META-INF/ejb-jar.xml WebSphere:cell=bpm85Cell1,node=AppSrv01Node,server=foobar ]] -BindJndiForEJBMessageBinding [[ SampleMDBEJB SampleMDB SampleMDBEJB.jar,META-INF/ejb-jar.xml "" jms/ESB101_AS jms/ESB101_Q wasadmin ]]]' ) 
AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

Restart Server

AdminControl.invoke('WebSphere:name=foobar,process=foobar,platform=proxy,node=AppSrv01Node,j2eeType=J2EEServer,version=8.5.5.2,type=Server,mbeanIdentifier=cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar/server.xml#Server_1416599363822,cell=bpm85Cell1,spec=1.0,processType=ManagedProcess', 'restart', '[]', '[]')

WMQ Cluster > MDB Testing

On ESB box

/opt/mqm/samp/bin/amqsput ESB101_Q ESB101_QM

Sample AMQSPUT0 start
target queue is ESB101_Q
Hello
WebSphere
From
WMQ

Sample AMQSPUT0 end

On BPM box

cat /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/logs/foobar/SystemOut.log

[22/11/14 07:57:30:958 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => Hello
[22/11/14 07:57:36:271 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => WebSphere
[22/11/14 07:57:38:475 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => From
[22/11/14 07:57:40:350 GMT] 0000009d SystemOut     O +++ SAMPLE MDB: Text Message => WMQ

Notes and Lessons Learned - 1

The credentials specified in the J2C Authentication Alias are crucial. At one point, I mis-specified the password, and saw the following exceptions in the WAS SystemOut.log : -

eMDBEJBEAR application is bound to the jms/ESB101_AS activation specification.
[22/11/14 08:01:38:857 GMT] 00000081 SibMessage    W   [:] CWSJY0003W: MQJCA4023: Startup reconnection failed for ActivationSpec 'javax.jms.Queue:jms/ESB101_Q@BPM855_QM <1648233785>'. Exception details: '
                       Message : com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'BPM855_QM' with connection mode 'Bindings' and host name 'localhost(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.

     Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED') reason '2035' ('MQRC_NOT_AUTHORIZED').

[22/11/14 08:01:38:864 GMT] 00000081 ActivationSpe E   J2CA0138E: The Message Endpoint activation failed for ActivationSpec jms/ESB101_AS (com.ibm.mq.connector.inbound.ActivationSpecImpl) and MDB application SampleMDBEJBEAR#SampleMDBEJB.jar#SampleMDB due to the following exception: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.

Caused by: com.ibm.msg.client.jms.DetailedJMSSecurityException: JMSWMQ2013: The security authentication was not valid that was supplied for QueueManager 'BPM855_QM' with connection mode 'Bindings' and host name 'localhost(1414)'.
Please check if the supplied username and password are correct on the QueueManager to which you are connecting.

[22/11/14 08:01:38:883 GMT] 00000081 RAWrapperImpl E   J2CA0089E: The method activateEndpoint on ResourceAdapter JavaBean cells/bpm85Cell1/nodes/AppSrv01Node/servers/foobar/resources.xml#J2CResourceAdapter_1416556034607 failed with the following exception: com.ibm.mq.connector.DetailedResourceAdapterInternalException: MQJCA1011: Failed to allocate a JMS connection. An internal error caused an attempt to allocate a connection to fail. See the linked exception for details of the failure.

with this in the WMQ log - /var/mqm/qmgrs/BPM855_QM/errors/AMQERR01.LOG  - for the local ( to WAS ) Queue Manager: -

AMQ5534: User ID 'wasadmin' authentication failed

EXPLANATION:
The user ID and password supplied by 'java' could not be authenticated.

AMQ5542: The failed authentication check was caused by the queue manager
CONNAUTH CHCKLOCL(OPTIONAL) configuration.

EXPLANATION:
The user ID 'wasadmin' and its password were checked because the queue manager
connection authority (CONNAUTH) configuration refers to an authentication
information (AUTHINFO) object named 'SYSTEM.DEFAULT.AUTHINFO.IDPWOS' with
CHCKLOCL(OPTIONAL).

I previously wrote about this here: -


in the context of the IBM Integration Bus Toolkit connecting to WMQ.

Bottom line, credentials are important :-)

Notes and Lessons Learned - 2

The biggest mistake I made was to assume that it was necessary to create the cluster Queue ( ESB101_Q of type QLOCAL ) on the ESB box. I spent many a fine hour trying to work out why things didn't work.

The point is that a client application ( amqsget or the MDB ) cannot connect to/use a Queue that's on a Queue Manager that's NOT local to the client. I'd assumed that I could make a bindings connection ( client to Queue Manager directly, without creating a client connection / channels etc. ) to a Queue Manager that's NOT local to the client/application.

This is why I do this job, every day is a school day, and I love to learn ...... :-)

Friday, 21 November 2014

DB2 Buffer Pools - Automagically tuning

So I picked this nugget of wisdom up this week.

One of my most excellent DB2 SME colleagues pointed out the wonderful db2top utility, specifically in the context of monitoring Buffer Pools.

You see, ever since I learned to install IBM Operational Decision Manager (ODM), I've been creating a Buffer Pool, bp32k, as required by the documentation and, more importantly, the product: -

db2 create bufferpool BP32K size 8000 pagesize 32 K

The DB2 SME, let's call him ... John, pointed out that a Buffer Pool of 8,000 pages x 32K may not always be large enough for one's requirements.

He showed me how db2top can be used to see this, as per the following example ( NOT from ODM, hence the different BP names ): -


In this example, we have 5 Buffer Pools, one of which IBMDEFAULTBP, is getting the most hits.

John pointed out that a Buffer Pool can be reconfigured to support automatic tuning, allowing it to grow IF the need arises.

Failure to grow means that the Buffer Pool is going to start paging to disk, and we do NOT want paging to occur.

So we can check what Buffer Pools we have in a database: -

db2 connect to bpmdb1

   Database Connection Information

 Database server        = DB2/LINUXX8664 10.1.3
 SQL authorization ID   = DB2INST1
 Local database alias   = BPMDB1

db2 "SELECT BPNAME, NPAGES, PAGESIZE FROM SYSCAT.BUFFERPOOLS"

BPNAME                                                                                                                           NPAGES      PAGESIZE   
-------------------------------------------------------------------------------------------------------------------------------- ----------- -----------
IBMDEFAULTBP                                                                                                                              -2       32768

  1 record(s) selected.


and we can then check whether automatic configuration is enabled: -

db2 "select BP_NAME,AUTOMATIC FROM TABLE(MON_GET_BUFFERPOOL('',-2))"

BP_NAME                                                                                                                          AUTOMATIC
-------------------------------------------------------------------------------------------------------------------------------- ---------
IBMDEFAULTBP                                                                                                                             1
IBMSYSTEMBP4K                                                                                                                            0
IBMSYSTEMBP8K                                                                                                                            0
IBMSYSTEMBP16K                                                                                                                           0
IBMSYSTEMBP32K                                                                                                                           0

  5 record(s) selected.


*IF* the Buffer Pool hadn't been set to auto-tune, we could've then changed it as follows: -

db2 "alter bufferpool IBMDEFAULTBP immediate size 1000 automatic"

Bottom line, db2top is your friend, and one should always follow the same process: -

Instrument > Test > Monitor > Tune > Test > Monitor > Tune ............