Finding the needle in the Deadlock haystack

I'm reposting this from an excellent blog post here: -

Finding the needle in the Deadlock haystack

<snip>

How many people out there have enjoyed the hunt for the deadlock culprits? If you are very analytical and enjoy a good murder mystery then perhaps you enjoy tracking down the two lines of application code that caused a deadlock that you wanted to resolve. For those that have not had the pleasure, let me start from the beginning…what is a deadlock?

In any database system a deadlock occurs when two applications are both waiting on a resource (row lock usually) that the other holds. Neither can proceed because they are waiting on each other and neither will voluntarily release their locks so you are in a deadlock. All database systems have a deadlock detector that wakes up on occasion, looks for deadlocks and if it finds one, it kills one of the transactions (rolls back the current unit of work) which results in that application releasing its lock so the other application can proceed. In the case of DB2 the application that is the victim gets a SQL0911 telling the app that the transaction was rolled back due to a deadlock or lock timeout.

</snip>

<snip>

With DB2 there is a deadlock event monitor that will dump out information when a deadlock occurs. In fact with DB2 v8.2, a DEADLOCKS WITH DETAILS event monitor is created automatically when the database is created and is started every time the database is started. This detailed deadlock monitor will dump information about the applications involved in any deadlock, including userids, hostnames of client machines, the statement text of the currently executing statement, and a list of all locks held by the application. The information is stored in the default database path (which you can find from your dbm configuration) in a directory called db2event/db2detaildeadlock.

</snip>

This from my own DB2 LUW environment: -

$ db2level

DB21085I  Instance "db2inst1" uses "64" bits and DB2 code release "SQL09075" 
with level identifier "08060107".
Informational tokens are "DB2 v9.7.0.5", "s111017", "IP23292", and Fix Pack 
"5".
Product is installed at "/opt/ibm/db2/V9.7".

$ db2evmon -db bpmdb  -evm db2detaildeadlock

Reading /home/db2inst1/db2inst1/NODE0000/SQL00003/db2event/db2detaildeadlock/00000000.evt ...
--------------------------------------------------------------------------
                            EVENT LOG HEADER
  Event Monitor name: DB2DETAILDEADLOCK
  Server Product ID: SQL09075
  Version of event monitor data: 11
  Byte order: LITTLE ENDIAN
  Number of nodes in db2 instance: 1
  Codepage of database: 1208
  Territory code of database: 1
  Server instance name: db2inst1
--------------------------------------------------------------------------

--------------------------------------------------------------------------
  Database Name: BPMDB   
  Database Path: /home/db2inst1/db2inst1/NODE0000/SQL00003/
  First connection timestamp: 29/08/2013 13:02:20.966657
  Event Monitor Start time:   29/08/2013 13:02:21.275886
--------------------------------------------------------------------------

--------------------------------------------------------------------------
  Database Name: BPMDB   
  Database Path: /home/db2inst1/db2inst1/NODE0000/SQL00003/
  First connection timestamp: 29/08/2013 13:07:52.540448
  Event Monitor Start time:   29/08/2013 13:08:10.467783
--------------------------------------------------------------------------

--------------------------------------------------------------------------
  Database Name: BPMDB   
  Database Path: /home/db2inst1/db2inst1/NODE0000/SQL00003/
  First connection timestamp: 29/08/2013 13:26:22.695516
  Event Monitor Start time:   29/08/2013 13:26:28.356203
--------------------------------------------------------------------------

--------------------------------------------------------------------------
  Database Name: BPMDB   
  Database Path: /home/db2inst1/db2inst1/NODE0000/SQL00003/
  First connection timestamp: 29/08/2013 13:31:07.766264
  Event Monitor Start time:   29/08/2013 13:31:08.846314
--------------------------------------------------------------------------

--------------------------------------------------------------------------
  Database Name: BPMDB   
  Database Path: /home/db2inst1/db2inst1/NODE0000/SQL00003/
  First connection timestamp: 29/08/2013 13:35:22.084183
  Event Monitor Start time:   29/08/2013 13:35:22.574086
--------------------------------------------------------------------------

--------------------------------------------------------------------------
  Database Name: BPMDB   
  Database Path: /home/db2inst1/db2inst1/NODE0000/SQL00003/
  First connection timestamp: 29/08/2013 13:35:22.756913
  Event Monitor Start time:   29/08/2013 13:35:23.409371
--------------------------------------------------------------------------

--------------------------------------------------------------------------
  Database Name: BPMDB   
  Database Path: /home/db2inst1/db2inst1/NODE0000/SQL00003/
  First connection timestamp: 29/08/2013 13:37:11.460381
  Event Monitor Start time:   29/08/2013 13:37:22.980311
--------------------------------------------------------------------------

IBM InfoSphere Data Architect V9.1

One of my colleagues was looking for a tool with which he could model some relational databases.

I asked my network, and found this: -

Evaluate: IBM InfoSphere Data Architect V9.1

IBM® InfoSphere Data Architect is a collaborative data design solution to help you discover, model, relate, and standardize diverse and distributed data assets. It supports dimensional modelling.

He downloaded the trial, installed it and was up and running …...

Using DB2TOP to work out what table is being hit by an IBM BPM operation

I was trying to ascertain which DB2 table was being affected by a particular operation in IBM BPM, that of creating an internal group: -

To be 100% clear, these are INTERNAL groups, rather than groups that are available to WebSphere Application Server, perhaps held in LDAP,  accessible via WebSphere Identity Manager (WIM) / Virtual Member Manager (VMM).

So, to come back to my core requirement, I was trying to work out what table(s) get updated when I create a new group ( the reason for this is that I've just created ~470 groups by hand, and wanted to query DB2 to check that I'd not missed any!!! ).

My underlying database is DB2 LUW, and I was looking for a way to monitor the database whilst I was using it within BPM.

Introducing DB2TOP  ….

This neat-o utility, probably based on Unix' top command provides a GUI ( albeit NOT X11-based ) view of the world.

I executed it as follows: -

$ db2top -d bpmdb

 

and then hit T ( capital T ) for Tables, rather than t ( lowercase t ) for Tablespaces, and then added a new internal group - SNAFU - and watched DB2TOP: -

This showed me that the table being updated was LSW_USR_GRP_XREF, which is precisely what I was looking for: -

$ db2 "select GROUP_NAME from db2inst1.LSW_USR_GRP_XREF" with UR

GROUP_NAME

...

FOOBAR

SNAFU

foobar

tw_admins

...

Some further reading: -

DB2 problem determination using db2top utility - Optimize performance and prevent problems in complex DB2 environments

Bad science ? Or just bad mathematics ? Or just me ?

From this BBC News article: -

University of St Andrews scientists create 'fastest man-made spinning object'

<snip>

A team of researchers claims to have created the world's fastest spinning man-made object.

They were able to levitate and spin a microscopic sphere at speeds of up to 600 million revolutions per minute.

This spin speed is half a million times faster than a domestic washing machine and more than a thousand times faster than a dental drill.

</snip>

For some reason, this piqued my interest.

The way that I read the two highlighted phrases is this: -

1) The researchers have an object that can spin at up to 600 million RPM ( 600,000,000 RPM )

2) That is 0.5 million ( 500,000 ) RPM faster than a domestic washing machine

3) Ergo, domestic washing machines spin at 599.5 million RPM ( 599,500,000 RPM )

According to another reputable ( in my personal opinion ) source, Which?

<snip>

Many washing machines boast a high 1400 revolutions per minute (rpm), 1600rpm or even 1800rpm spin speed rather than the more common 1200rpm, which helps save time on the line or energy use in the tumble dryer by removing more water. However you'll usually pay extra for this feature, and 1200rpm is generally adequate for most loads.

</snip>

So the BBC are comparing this lab experiment with a washing machine … that can spin at 599.5 million RPM.

Can someone please explain the error in my working ? 'Cos I am confused.com

IBM BPM 8.5 - My first exposure ....

Hmmmm, over two months since it was released, and I finally made the time to download IBM BPM 8.5.

I've not yet installed it, but I'm looking forward to the process.

In the meantime, this is what I downloaded: -

IBM Business Process Manager Advanced Version 8.5 Linux X86 32Bit /X86 64Bit Multilingual eAssembly (CRMM5ML)

or, more specifically: -

IBM Business Process Manager Advanced Version 8.5 For Linux X86 32/64Bit Multilingual - 1 of 3 (CIL96ML) 
IBM Business Process Manager Advanced Version 8.5 For Linux X86 32/64Bit Multilingual - 2 of 3 (CIL97ML) 

IBM Business Process Manager Advanced Version 8.5 For Linux X86 32/64Bit Multilingual - 3 of 3 (CIL98ML) 

as I already have WAS 8.5 and DB2 10, and won't use LDAP in the first instance.

I do have to remember that I need WAS 8.5.0.2 ( aka 8.5.0 Fixpack 2 ) and DB2 10.1 Fixpack 1.

Download IBM Business Process Manager Advanced Version 8.5 (Linux)

What fun :-)

From the Twitter stream - IBM web server plug-in is not load balancing evenly

Problem(Abstract)

IBM web server plug-in appears to not be load balancing correctly. Too much load is going to only one of the members (application servers) in the cluster. Why is this happening?

Symptom

During load testing, you might notice that a lot more work is being handled by only 1 application server in the cluster. However, the other application servers in the same cluster are much less busy by comparison.

Cause

The IBM web server plug-in only load balances non-affinity requests. These are new requests that do not have a session cookie (JSESSIONID) in the request.

On the other hand, if it is an affinity request, that does have a session cookie (JSESSIONID), then the plug-in will honor session affinity by sending the request to the same application server as before. It will use the application server which matches the cloneid in the JSESSIONID cookie.

Want to know more ? Then read the IBM Technote 1449249 here: -

IBM web server plug-in is not load balancing evenly

IBM Connections Connector for Apache OpenOffice

So it's been waaaaaaay too long since I posted about IBM Connections, even though it's a tool that I use each and every day.

I saw this on my "river of streaming news" this morning: -

IBM Connections Connector for Apache OpenOffice

IBM Connections connector 2.0 is now available for download. It can help you connect to IBM Connections server to access and manage your files and other resources, while you are still working on the OpenOffice. The innovative and popular feature of Sidebar in OpenOffice 4.0 has been brought here to provide you a better experience and usability.

2.0
This release is updated for the Apache OpenOffice 4.0. It provides a new UI which is fully integrated with the new feature of sidebar, which makes better use of today's widescreen and can let you access your files more easily.
This release will focus on connection with Files of IBM Connections. Other services e.g. status update will be supported in the following release.
Windows, Mac and Linux are all supported.

- See more at: http://extensions.openoffice.org/en/project/ibm-connections-connector-apache-openoffice#sthash.ZfFymCgi.dpuf

IBM Connections Connector for Apache OpenOffice

Will get on and download this …...

IBM HTTP Server - Disabling the HTTP TRACE method

One of my colleagues contacted me yesterday asking whether it was possible to disable the HTTP TRACE method in IBM HTTP Server.

He referenced me to this page in the IHS Information Center: -

Disabling the HTTP TRACE method

http://publib.boulder.ibm.com/httpserv/ihsdiag/http_trace.html

which says, in part: -

The HTTP TRACE request method causes the data received by IBM HTTP Server from the client to be sent back to the client, as in the following example:

$ telnet 127.0.0.1 8080

Trying...

Connected to 127.0.0.1.

Escape character is '^]'.

TRACE / HTTP/1.0

Host: foo

A: b

C: d

HTTP/1.1 200 OK

Date: Mon, 04 Oct 2004 14:07:59 GMT

Server: IBM_HTTP_SERVER

Connection: close

Content-Type: message/http

TRACE / HTTP/1.0

A: b

C: d

Host: foo

Connection closed.

The TRACE capability could be used by vulnerable or malicious applications to trick a web browser into issuing a TRACE request against an arbitrary site and then send the response to the TRACE to a third party using web browser features.

He didn't have an IHS server with which to play, so I tested the hypothesis on my own Red Hat Enterprise Linux box, using IHS 7.0.0.0: -

With TRACE enabled in HTTPD.CONF ( default state )

$ telnet rhel6.uk.ibm.com 8080

Trying 127.0.0.1...

Connected to rhel6.uk.ibm.com.

Escape character is '^]'.

TRACE / HTTP/1.0

Host: foo

A: b

C: d

HTTP/1.1 200 OK

Date: Wed, 21 Aug 2013 10:17:58 GMT

Server: IBM_HTTP_Server

Connection: close

Content-Type: message/http

TRACE / HTTP/1.0

Host: foo

A: b

C: d

Connection closed by foreign host.

With TRACE disabled in HTTPD.CONF

I added: -

LoadModule rewrite_module modules/mod_rewrite.so

RewriteEngine On 

RewriteCond %{REQUEST_METHOD} ^TRACE 

RewriteRule .* - [F] 

to httpd.conf and restarted IHS ( /opt/IBM/HTTPServer/lbin/apachectl start )

$ telnet rhel6.uk.ibm.com 8080

Trying 127.0.0.1...

Connected to rhel6.uk.ibm.com.

Escape character is '^]'.

TRACE /HTTP/1.0

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

<html><head>

<title>403 Forbidden</title>

</head><body>

<h1>Forbidden</h1>

<p>You don't have permission to access /HTTP/1.0

on this server.</p>

<hr>

<address>IBM_HTTP_Server at localhost Port 8080</address>

</body></html>

Connection closed by foreign host.

which looks to be a good sign.

The access.log file also confirms the behaviour: -

127.0.0.1 - - [21/Aug/2013:11:17:58 +0100] "TRACE / HTTP/1.0" 200 43 <- This is what I see BEFORE I changed HTTPD.CONF and restarted IHS
127.0.0.1 - - [21/Aug/2013:11:32:07 +0100] "TRACE /HTTP/1.0" 403 273 <- This is what I see AFTER changing HTTPD.CONF and restarting IHS

We also "spoke" about the the HTTP TRACK method. Again, the Information Center article says this: -

The HTTP TRACK method

The TRACK method is a type of request supported by Microsoft web servers. It is not RFC compliant and is not supported directly by IBM HTTP Server. The method may be utilized as part of a cross-site scripting attack. See Vulnerability Note VU#288308 for more information.

Even though IBM HTTP Server does not support the TRACK method natively, it is possible for plug-in modules to provide support for it. To disable this capability for plug-in modules, in addition to disabling the TRACE method, add these two additional directives after the existing RewriteCond and RewriteRule directives which are used to disable TRACE:

RewriteCond %{REQUEST_METHOD} ^TRACK

RewriteRule .* - [F]

Here is a full example showing the directives to disable both TRACE and TRACK:

...

# disable TRACE and TRACK in the main scope of httpd.conf

RewriteEngine On 

RewriteCond %{REQUEST_METHOD} ^TRACE 

RewriteRule .* - [F] 

RewriteCond %{REQUEST_METHOD} ^TRACK

RewriteRule .* - [F] 

...

<VirtualHost www.example.com>

...

# disable TRACE and TRACK in the www.example.com virtual host

RewriteEngine On 

RewriteCond %{REQUEST_METHOD} ^TRACE 

RewriteRule .* - [F] 

RewriteCond %{REQUEST_METHOD} ^TRACK

RewriteRule .* - [F] 

</VirtualHost>

Note that for IBM HTTP Server 7.0 and later, the only method to disable TRACK is via mod_rewrite. Since IHS doesn't do anything with TRACK, there is no directive to "disable" it.

So, for completeness, here's HTTPD.CONF now: -

<snip>
LoadModule rewrite_module modules/mod_rewrite.so

RewriteEngine On 
RewriteCond %{REQUEST_METHOD} ^TRACE 
RewriteRule .* - [F] 
RewriteCond %{REQUEST_METHOD} ^TRACK
RewriteRule .* - [F]

</snip>

Sweet :-)

Finally, I also recommended that my colleague review ( and share ) this excellent developerWorks article by Martin Lansche and Keys Botzum: -

Advanced security hardening in WebSphere Application Server V7, V8 and V8.5, Part 1: Overview and approach to security hardening

Summary:  Security consists of more than just some firewalls at the edge of your network protecting you from the outside. It is a difficult and complex set of actions and procedures that strive to strengthen your systems as much as is appropriate. This article discusses many aspects of security in general, including the IBM® WebSphere® Application Server security architecture, and discusses hardening a WebSphere Application Server environment. This updated article has been significantly revised for WebSphere Application Server V7, 8.0, and 8.5, and has been edited to focus solely on hardening. Part 1 of 2. This content is part of the IBM WebSphere Developer Technical Journal.

IBM BPM - Process Center to Process Server - How is the authentication handled ?

We're currently configuring a newly built Process Center / Process Server environment, and wondered how PC authenticates to PS in order to allow a developer to install a snapshot.

We're using BPM Standard 7.5.1.1.

This is configured, on the Process Server box, in 100Custom.xml as follows: -

…

    <repository-server-url>https://process-center.uk.ibm.com:9443/ProcessCenter</repository-server-url>;

    <repository-server-user-auth-alias>BPMRuntimeServer_Auth_Alias</repository-server-user-auth-alias>

    <repository-server-designated-user-auth-alias>BPMAuthor_Auth_Alias</repository-server-designated-user-auth-alias>

    <repository-server-interval>10</repository-server-interval>
    <!-- Force Process Center Server to use https to deploy ProcessApps and Toolkits to Process Servers -->
    <!--
        <deploy-snapshot-using-https>true</deploy-snapshot-using-https>
        -->
    <server-name>ProcessServer</server-name>
    <server-description>A running process server</server-description>
    <server-host>process-server.uk.ibm.com</server-host>
    <server-port>9443</server-port>

…

The aliases can be seen via the WAS Integrated Solutions Console, under Security > Global security > JAAS - J2C authentication data > BPMAuthor_Auth_Alias, as per the example: -

The IBM Information Center covers this in far more detail: -

Modifying IBM Process Server connection properties

Optional: Edit the <repository-server-user-auth-alias> and <repository-server-designated-user-auth-alias> property values. The <repository-server-user-auth-alias> value specifies the authentication alias for a user to connect from the process server to the Process Center. The user and password set in this alias must be present in the Process Center.

The <repository-server-designated-user-auth-alias> value specifies the authentication alias for a user to access and deploy snapshots to the runtime process server and access that process server from the Process Inspector, which is located in IBM Process Designer. This authentication alias must be defined in both that process server and the Process Center and the passwords must match.

CWLLG1162E: An exception occurred while setting the attribute Locale to value en for user wasadmin

Doh, twice this week, I've seen these exceptions in my IBM BPM Standard 7.5.1.1 server's SystemOut.log ( this is for the AppTarget cluster ): -

[15/08/13 17:33:40:004 BST] 00000031 wle_security  E   CWLLG1162E: An exception occurred while setting the attribute Locale to value en for user wasadmin. Error: com.lombardisoftware.client.delegate.BusinessDelegateException: java.lang.NullPointerException
                                 com.lombardisoftware.client.delegate.BusinessDelegateException: java.lang.NullPointerException
        at com.lombardisoftware.client.delegate.BusinessDelegateException.asBusinessDelegateException(BusinessDelegateException.java:41)
        at com.lombardisoftware.client.delegate.PersistenceServicesDelegateDefault.findSingleByFilter(PersistenceServicesDelegateDefault.java:251)

[15/08/13 17:33:40:076 BST] 00000031 wle           E   CWLLG2229E: An exception occurred in an EJB call.  Error: java.lang.NullPointerException
                                 com.lombardisoftware.client.delegate.BusinessDelegateException: java.lang.NullPointerException
        at com.lombardisoftware.client.delegate.BusinessDelegateException.asBusinessDelegateException(BusinessDelegateException.java:41)
        at com.lombardisoftware.client.delegate.PersistenceServicesDelegateDefault.findSingleByFilter(PersistenceServicesDelegateDefault.java:251)

[15/08/13 17:33:40:095 BST] 00000031 servlet       E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0068E: Uncaught exception created in one of the service methods of the servlet /repository/com.lombardisoftware.repository.Repository/Repository.jsp in application IBM_BPM_Repository_ProcessCenter.AppTarget. Exception created : com.ibm.websphere.servlet.error.ServletErrorReport: com.lombardisoftware.client.delegate.BusinessDelegateException: java.lang.NullPointerException

Each time, I spend a manic few moments trying to work out what\s gone wrong.

And then I remember …..

I did NOT bootstrap the database tables: -

$ cd /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin
$ ./bootstrapProcessServerData.sh 

Can you say "Doh!" ?

PS The reason that I'm doing this over and again is that I've been cleaning down my environment ( stopping JVMs, deleting WAS profiles, dropping the BPMDB/PDWDB/MEDB databases etc. ) multiple times, whilst I test the application of a series of iFixes.

SRVE0255E: A WebGroup/Virtual Host to handle / has not been defined.

Another warning that we see in our logs in HUGE quantities is: -

SRVE0255E: A WebGroup/Virtual Host to handle / has not been defined.

This occurs, for us, because our WebSEAL secure reverse proxy servers ( WebSEAL is part of the IBM Tivoli Access Manager for e-Business authentication/authorisation/access management solution ).

It looks like WebSEAL regularly polls the IBM BPM servers, but "merely" requests the / context root e.g.

http://processserver.uk.ibm.com:9080/

rather than: _

http://processserver.uk.ibm.com:9080/portal

and the underlying WAS does not have a Virtual Host set up for /, only for context roots such as: -

/portal

/ProcessCenter

/ProcessAdmin

etc.

Given that we have a large farm of WebSEAL servers sitting in front of IBM BPM, we see the message really rather regularly.

It's another "error" that we can safely ignore, but it's a bit of a niggle.

CWLLG2101W: Retries were explicitly requested in ProgrammaticTransactionSupport but no new transaction is started

We've been seeing LOADS of these warnings in our logs: -

[16/08/13 08:26:39:608 BST] 00000054 wle           W   CWLLG2101W: Retries were explicitly requested in ProgrammaticTransactionSupport but no new transaction is started 5 definition: PROPAGATION_REQUIRED,ISOLATION_DEFAULT retryLimit:{2}

As a warning, I had assumed that it was mostly harmless.

This IBM Technote confirms that: -

A CWLLG2101W message indicates that new transactions are not started for IBM Business Process Manager (BPM)

…

This message is harmless.  It is currently showing up in levels of trace where it should not.

Further information: This message is intended to be internal to the components that are using transaction support. If one component calls another component within a transaction context and it specifies a number of retries greater than 0, the message is generated.

…

So it's a niggle, and I wish I didn't appear, but at least I / we can filter it out.

Using IBM Installation Manager to report on available iFixes for IBM BPM

Another post in the "If I don't write it down, I'll forget it" tradition of this blog: -

So I have a bunch of iFixes downloaded as .ZIP files: -

-rw-r--r--  1 wasadmin wasadmins   142K Aug 12 13:15 7.5.1.1-WS-BPM-IFJR45071.zip
-rw-r--r--  1 wasadmin wasadmins   1.8M Aug 12 13:15 7.5.1.1-WS-BPM-IFJR44032.zip
-rw-r--r--  1 wasadmin wasadmins   189K Aug 12 13:15 7.5.1.1-WS-BPM-IFJR45553.zip
-rw-r--r--  1 wasadmin wasadmins   144K Aug 12 13:15 7.5.1.1-WS-BPM-IFJR45523.zip
-rw-r--r--  1 wasadmin wasadmins    26M Aug 12 13:15 7.5.1.1-WS-BPM-IFJR45124.zip
-rw-r--r--  1 wasadmin wasadmins   615K Aug 12 13:15 7.5.1.1-WS-BPM-IFJR44463.zip
-rw-r--r--  1 wasadmin wasadmins    25M Aug 12 13:15 7.5.1.1-WS-BPM-IFJR45344.zip
-rw-r--r--  1 wasadmin wasadmins   613K Aug 12 13:15 7.5.1.1-WS-BPM-IFJR44086.zip
-rw-r--r--  1 wasadmin wasadmins    16M Aug 12 13:15 7.5.1.1-WS-BPM-IFJR44606.zip
-rw-r--r--  1 wasadmin wasadmins   173K Aug 12 13:15 7.5.1.1-WS-BPM-Unix-IFJR43774.zip
-rw-r--r--  1 wasadmin wasadmins    40M Aug 12 13:15 7.5.1.1-WS-BPM-IFJR44617.zip
-rw-r--r--  1 wasadmin wasadmins   261K Aug 12 13:15 7.5.1.1-WS-BPM-IFJR45755.zip
-rw-r--r--  1 wasadmin wasadmins   261K Aug 12 13:15 7.5.1.1-WS-BPM-IFJR45931.zip
-rw-r--r--  1 wasadmin wasadmins    54M Aug 12 13:15 7.5.1.1-WS-BPM-IFJR46350.zip
-rw-r--r--  1 wasadmin wasadmins   1.5M Aug 12 13:15 7.5.1.1-WS-BPM-IFIC86727.zip
-rw-r--r--  1 wasadmin wasadmins    25M Aug 12 13:31 7.5.1.1-WS-BPM-IFJR45222.zip
-rw-r--r--  1 wasadmin wasadmins    16M Aug 12 13:31 7.5.1.1-WS-BPM-IFJR45851.zip
-rw-r--r--  1 wasadmin wasadmins   145K Aug 12 13:31 7.5.1.1-WS-BPM-IFJR44474.zip
-rw-r--r--  1 wasadmin wasadmins   288K Aug 12 13:31 7.5.1.1-WS-BPM-IFJR44199.zip
-rw-r--r--  1 wasadmin wasadmins   329K Aug 12 13:38 7.5.1.1-WS-BPM-IFJR44357.zip
-rw-r--r--  1 wasadmin wasadmins   259K Aug 12 13:38 7.5.1.1-WS-BPM-IFJR44690.zip
-rw-r--r--  1 wasadmin wasadmins   439K Aug 12 13:41 7.5.1.1-WS-BPM-IFJR42774.zip
-rw-r--r--  1 wasadmin wasadmins   158K Aug 12 16:24 7.5.1.1-WS-BPM-IFJR44428.zip
-rw-r--r--  1 wasadmin wasadmins   185K Aug 12 16:24 7.5.1.1-WS-BPM-IFJR44397.zip
-rw-r--r--  1 wasadmin wasadmins   138K Aug 12 16:24 7.5.1.1-WS-BPM-IFJR44503.zip
-rw-r--r--  1 wasadmin wasadmins    25M Aug 12 16:24 7.5.1.1-WS-BPM-IFJR45532.zip
-rw-r--r--  1 wasadmin wasadmins   246K Aug 15 13:56 7.5.1.1-WS-BPM-IFJR45939.zip
-rw-r--r--  1 wasadmin wasadmins   156K Aug 15 13:56 7.5.1.1-WS-BPM-IFJR44729.zip

I've unzipped each of these into their own discrete directory: -

drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFIC86727
drwxr-xr-x  6 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR44032
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR44086
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR44463
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR44606
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR44617
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR45071
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR45124
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR45344
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR45523
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR45553
drwxr-xr-x  6 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR45755
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR45931
drwxr-xr-x  6 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-IFJR46350
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:22 7.5.1.1-WS-BPM-Unix-IFJR43774
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:34 7.5.1.1-WS-BPM-IFJR45222
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:34 7.5.1.1-WS-BPM-IFJR45851
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:35 7.5.1.1-WS-BPM-IFJR44474
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:35 7.5.1.1-WS-BPM-IFJR44199
drwxr-xr-x  6 wasadmin wasadmins 4.0K Aug 12 13:39 7.5.1.1-WS-BPM-IFJR44357
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 13:39 7.5.1.1-WS-BPM-IFJR44690
drwxr-xr-x  6 wasadmin wasadmins 4.0K Aug 12 13:41 7.5.1.1-WS-BPM-IFJR42774
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 16:25 7.5.1.1-WS-BPM-IFJR44428
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 16:25 7.5.1.1-WS-BPM-IFJR44397
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 16:26 7.5.1.1-WS-BPM-IFJR44503
drwxr-xr-x  3 wasadmin wasadmins 4.0K Aug 12 16:26 7.5.1.1-WS-BPM-IFJR45532
drwxr-xr-x  7 wasadmin wasadmins 4.0K Aug 15 09:57 7.5.1.1-WS-BPM-IFJR44729
drwxr-xr-x  6 wasadmin wasadmins 4.0K Aug 15 13:57 7.5.1.1-WS-BPM-IFJR45939

Want to know how I did it ?

Read this post: -

Unzipping IBM iFixes in a FOR loop

Having got my fixes in order, I can now use IBM Installation Manager to report on what's available: -

$ for i in `find ~/Fixes/Install/ | grep repository.xml`; do /opt/IBM/InstallationManager/eclipse/tools/imcl listAvailableFixes com.ibm.ws.WBPMSTD_7.5.1001.20120915_1227 -repositories $i; done

which returns: -

7.5.1.1-WS-BPM-IFJR45124_0.0.0.20130121_1508
7.5.1.1-WS-BPM-IFJR44474_0.0.0.20121203_2247
7.5.1.1-WS-BPM-IFJR43774_0.0.0.20121113_1959
7.5.1.1-WS-BPM-IFJR44357_0.0.0.20130104_1511
7.5.1.1-WS-BPMPC-IFPD44357_0.0.0.20130104_1533
7.5.1.1-WS-BPM-IFIC86727_0.0.0.20130329_1242
7.5.1.1-WS-BPM-IFJR44729_0.0.0.20121116_1736
7.5.1.1-WS-BPM-IFJR45939_0.0.0.20130710_0014
7.5.1.1-WS-BPMPC-IFPD45939_0.0.0.20130711_1121
7.5.1.1-WS-BPM-IFJR45523_0.0.0.20130128_1539
7.5.1.1-WS-BPM-IFJR45222_0.0.0.20130123_0131
7.5.1.1-WS-BPM-IFJR45532_0.0.0.20130126_0434
7.5.1.1-WS-BPM-IFJR45553_0.0.0.20130131_1507
7.5.1.1-WS-BPM-IFJR44463_0.0.0.20121221_1452
7.5.1.1-WS-BPM-IFJR44086_0.0.0.20121206_2324
7.5.1.1-WS-BPM-IFJR44032_0.0.0.20130308_1342
7.5.1.1-WS-BPMPC-IFPD44032_0.0.0.20130308_1349
7.5.1.1-WS-BPM-IFJR45344_0.0.0.20130116_2314
7.5.1.1-WS-BPM-IFJR44617_0.0.0.20121101_1346
7.5.1.1-WS-BPM-IFJR42774_0.0.0.20121016_1300
7.5.1.1-WS-BPMPC-IFPC42774_0.0.0.20121126_1259
7.5.1.1-WS-BPMPC-IFPD42774_0.0.0.20121126_1517
7.5.1.1-WS-BPM-IFJR44397_0.0.0.20121106_1740
7.5.1.1-WS-BPM-IFJR45755_0.0.0.20130304_1154
7.5.1.1-WS-BPMPC-IFPD45755_0.0.0.20130220_1405
7.5.1.1-WS-BPM-IFJR44606_0.0.0.20121108_1605
7.5.1.1-WS-BPM-IFJR44428_0.0.0.20121024_1447
7.5.1.1-WS-BPM-IFJR44199_0.0.0.20130104_2055
7.5.1.1-WS-BPM-IFJR45071_0.0.0.20121218_1416
7.5.1.1-WS-BPM-IFJR46350_0.0.0.20130430_1238
7.5.1.1-WS-BPMPC-IFPC46350_0.0.0.20130424_1355
7.5.1.1-WS-BPMPC-IFPD46350_0.0.0.20130429_1544
7.5.1.1-WS-BPM-IFJR44690_0.0.0.20121108_1616
7.5.1.1-WS-BPM-IFJR44503_0.0.0.20121026_1403
7.5.1.1-WS-BPM-IFJR45931_0.0.0.20130523_1037
7.5.1.1-WS-BPM-IFJR45851_0.0.0.20130423_2340

Unzipping IBM iFixes in a FOR loop

I have a load of iFixes downloaded as .ZIP files: -

7.5.1.1-WS-BPM-IFIC86727.zip  7.5.1.1-WS-BPM-IFJR44463.zip  7.5.1.1-WS-BPM-IFJR45124.zip  7.5.1.1-WS-BPM-IFJR45931.zip
7.5.1.1-WS-BPM-IFJR42774.zip  7.5.1.1-WS-BPM-IFJR44474.zip  7.5.1.1-WS-BPM-IFJR45222.zip  7.5.1.1-WS-BPM-IFJR45939.zip
7.5.1.1-WS-BPM-IFJR44032.zip  7.5.1.1-WS-BPM-IFJR44503.zip  7.5.1.1-WS-BPM-IFJR45344.zip  7.5.1.1-WS-BPM-IFJR46350.zip
7.5.1.1-WS-BPM-IFJR44086.zip  7.5.1.1-WS-BPM-IFJR44606.zip  7.5.1.1-WS-BPM-IFJR45523.zip  7.5.1.1-WS-BPM-Unix-IFJR43774.zip
7.5.1.1-WS-BPM-IFJR44199.zip  7.5.1.1-WS-BPM-IFJR44617.zip  7.5.1.1-WS-BPM-IFJR45532.zip 
7.5.1.1-WS-BPM-IFJR44357.zip  7.5.1.1-WS-BPM-IFJR44690.zip  7.5.1.1-WS-BPM-IFJR45553.zip 
7.5.1.1-WS-BPM-IFJR44397.zip  7.5.1.1-WS-BPM-IFJR44729.zip  7.5.1.1-WS-BPM-IFJR45755.zip
7.5.1.1-WS-BPM-IFJR44428.zip  7.5.1.1-WS-BPM-IFJR45071.zip  7.5.1.1-WS-BPM-IFJR45851.zip

in a directory called /home/wasadmin/Fixes.

Here's a neat little script that I use to expand the fixes, neatly ready to be installed using IBM Installation Manager: -

$ cd /home/wasadmin/Fixes

$ mkdir Install

$ for i in *.zip;do unzip $i -d Install/`echo $i | sed -s 's/.zip//g'`; done

This results in each iFix unzipped into its own private directory: -

drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFIC86727
drwxr-xr-x  6 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR42774
drwxr-xr-x  6 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44032
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44086
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44199
drwxr-xr-x  6 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44357
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44428
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44397
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44463
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44474
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44503
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44606
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44617
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44690
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR44729
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45071
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45124
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45222
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45344
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45523
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45532
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45553
drwxr-xr-x  6 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45755
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45851
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45931
drwxr-xr-x  6 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR45939
drwxr-xr-x  6 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-IFJR46350
drwxr-xr-x  3 wasadmin wasadmins 4096 Aug 15 14:20 7.5.1.1-WS-BPM-Unix-IFJR43774

ready for installation using a response file.

SSL and WebSphere Application Server

The first of, almost certainly, a whole slew of posts as I dig into SSL in WAS 7.

I'm really trying to establish why my newly minted WAS profiles turn up with the cell-level and node-level root Signer certificate issuer set to: -

CN=localhost, OU=Root Certificate, OU=localhostNode01Cell, OU=rhel6Node01, O=IBM, C=US  

even though my fully-qualified hostname is: -

rhel6.uk.ibm.com

as confirmed by Java: -

import java.net.InetAddress;
import java.net.UnknownHostException;

public class getHost {

  public static void main(String[] args) {

    try {
      InetAddress address = InetAddress.getLocalHost();
      System.out.println("My name is " + address.getHostName());
    } catch (UnknownHostException e) {
      System.out.println("I'm sorry. I don't know my own name.");
    }

  }

}

Source: Java Tutorial » Network » Internet Addresses > 19.7.3.Get Host name from InetAddress

which returns: -

My name is rhel6.uk.ibm.com

and Snoop: -

Of course, these are self-signed certificates, and WAS is the signer / root Certificate Authority (CA).

But why does the Common Name (CN) come out as localhost rather than, say rhel6.uk.ibm.com ??

This is what I have in my node-level trust store: -

${CONFIG_ROOT}/cells/localhostNode01Cell/nodes/rhel6Node01/trust.p12

For the record, this particular profile is a pure application server ( AppSrv01 ) with no managed cell.

I did find this command: -

$ /opt/IBM/WebSphere/AppServer/bin/retrieveSigners.sh -listRemoteKeyStoreNames

Realm/Cell Name: <default>
Username: wasadmin
Password:         
 CWPKI0306I: The following remote keystores exist on the specified server:
           NodeDefaultRootStore, NodeDefaultKeyStore, NodeRSATokenRootStore,
           NodeRSATokenTrustStore, NodeDefaultSignersStore,
           NodeDefaultDeletedStore, NodeDefaultTrustStore,
           NodeRSATokenKeyStore, NodeLTPAKeys

which is useful, but doesn't solve the problem :-)

Anyhow, this is how I created the profile: -

-create
 -applyPerfTuningSetting standard
 -adminUserName wasadmin
 -adminPassword ****************
 -enableAdminSecurity true
 -cellName localhostNode01Cell
 -nodeName rhel6Node01
 -portsFile /opt/IBM/WebSphere/AppServer/logs/manageprofiles/1376559474506_portdef.props
 -personalCertDN cn=localhost, ou=localhostNode01Cell, ou=rhel6Node01, o=IBM, c=US
 -profilePath /opt/IBM/WebSphere/AppServer/profiles/AppSrv01
 -serverName server1
 -signingCertDN, cn=localhost, ou=Root Certificate, ou=localhostNode01Cell, ou=rhel6Node01, o=IBM, c=US
 -hostName rhel6.uk.ibm.com
 -profileName AppSrv01
 -personalCertValidityPeriod 15
 -signingCertValidityPeriod 15
 -keyStorePassword ****************
 -templatePath /opt/IBM/WebSphere/AppServer/profileTemplates/default

NOTE I used Profile Management Tool (PMT) to do this, and that automatically populated the personalCertDN and signingCertDN values with localhost.

I can, and will, override these next time around, but I'm wondering where PMT gets localhost from.

IBM BPM - Checking the precise version of IBM Process Designer

As ever, IBM Installation Manager has the answer, IBM Installation Manager is your friend ….

Here's the GUI: -

Here's the command-line: -

C:\> "C:\IBM\Installation Manager\eclipse\tools\imcl.exe" listInstalledPackages -long

...
C:\IBM\ProcessDesigner\v7.5 : com.ibm.bpm.authoring.v75_7.5.1001.v20120914_0614 : IBM? Process Designer : 7.5.1.1
C:\IBM\ProcessDesigner\v7.5 : JR42774_0.0.0.20120925_1011 : 7.5.1.1-WS-BPMPD-IFJR42774 : 0.0.0.20120925_1011
C:\IBM\ProcessDesigner\v7.5 : JR44032_0.0.0.20120925_1001 : 7.5.1.1-WS-BPMPD-IFJR44032 : 0.0.0.20120925_1001
C:\IBM\ProcessDesigner\v7.5 : JR44357_0.0.0.20121126_1531 : 7.5.1.1-WS-BPMPD-IFJR44357 : 0.0.0.20121126_1531
C:\IBM\ProcessDesigner\v7.5 : JR45755_0.0.0.20121217_1010 : 7.5.1.1-WS-BPMPD-IFJR45755 : 0.0.0.20121217_1010
…

What this shows me is that some of the recently applied iFixes also update IBM Process Designer.

It also shows me that I need to update Process Designer again, as there's an iFix ( JR44729 ) on the Process Center, that's not currently showing up in Process Designer.

This is the version of Process Designer that I'd previously downloaded from the Process Center, prior to applying JR44729: -

-rw-r--r--@  1 hayd  staff  424501873 13 Aug 16:41 IBM Process Designer.zip

This is the version that I have since downloaded, having installed JR44729: -

-rw-r--r--@  1 hayd  staff  424501713 15 Aug 08:58 IBM Process Designer.zip

Interestingly, it's slightly smaller ….

Having uninstalled Process Designer, and then re-installed it from the newly downloaded ( and smaller ) ZIP file, I re-ran my query: -

C:\> "C:\IBM\Installation Manager\eclipse\tools\imcl.exe" listInstalledPackages -long

...

C:\IBM\ProcessDesigner\v7.5 : com.ibm.bpm.authoring.v75_7.5.1001.v20120914_0614 : IBM? Process Designer : 7.5.1.1
C:\IBM\ProcessDesigner\v7.5 : JR42774_0.0.0.20120925_1011 : 7.5.1.1-WS-BPMPD-IFJR42774 : 0.0.0.20120925_1011
C:\IBM\ProcessDesigner\v7.5 : JR44032_0.0.0.20120925_1001 : 7.5.1.1-WS-BPMPD-IFJR44032 : 0.0.0.20120925_1001
C:\IBM\ProcessDesigner\v7.5 : JR44357_0.0.0.20121126_1531 : 7.5.1.1-WS-BPMPD-IFJR44357 : 0.0.0.20121126_1531
C:\IBM\ProcessDesigner\v7.5 : JR45755_0.0.0.20121217_1010 : 7.5.1.1-WS-BPMPD-IFJR45755 : 0.0.0.20121217_1010

...

So what changed ? Apart from the size of the ZIP file ??

Apparently, nothing.

This is despite the fact that the Readme.txt for JR44729 says: -

…

After installing fixes on Process Center, updated Process Designer needs to be downloaded and installed else the Process Designer will be out of sync with Process Center. See post install instructions below.

…

…

6. Post Install Instructions:

(After starting servers)

a) Start the stand-alone server or network deployment environment.

b) Launch Process Center console on any/all environments where Process Designer was installed.                                              

c) Download updated Process Designer image by clicking the 'Download Process Designer' button on the right of the Process center console

d) Run installProcessDesigner.bat specifying the path of the current install location if different then default location. This will update the existing Process Designer install with the above fixes.

…

So, given that JR44729 appears to have absolutely nothing to do with Process Designer: -

JR44729: P.E. UNABLE TO ASSIGN TEAM MANAGER GROUP FOR LDAP GROUPS.

...

Unable to add team manger group to ldap groups in process admin

console.

Logs show the following error PreparedStatementCallback; SQL

[insert into LSW_USR_GRP_XREF

(GROUP_ID,GROUP_NAME,DISPLAY_NAME,GROUP_TYPE,DESCRIPTION,GROUP_S

TATE,PARENT_GROUP_ID) values (?,?,?,?,?,?,?)]; Error for batch

element #1: DB2 SQL Error: SQLCODE=-803, SQLSTATE=23505,

SQLERRMC=2;BPMADMIN.LSW_USR_GRP_XREF, DRIVER=4.12.55; nested

exception is

com.ibm.db2.jcc.am.SqlIntegrityConstraintViolationException

This same problem might arise when you upgrade from TeamWorks

6.x to Business Process Manager 7.5.1.1.

…

I'm at a loss.

Perhaps a post-prandial PMR …..

*UPDATE* We discovered that we were missing a requisite iFix, IFJR45939 ,which also updates Process Designer.  This is required because we've applied a Recommended iFix, IFJR42774,

Therefore, I had to go back through another update of PD on my Windows VM.

This was the newly downloaded PD .ZIP file: -

-rw-r--r--@   1 hayd  staff  424609142 15 Aug 17:45 IBM Process Designer.zip

and this is what IBM Installation Manager now shows us in terms of the PD installation: -

Before

C:\IBM\ProcessDesigner\v7.5 : com.ibm.bpm.authoring.v75_7.5.1001.v20120914_0614 : IBM? Process Designer : 7.5.1.1
C:\IBM\ProcessDesigner\v7.5 : JR42774_0.0.0.20120925_1011 : 7.5.1.1-WS-BPMPD-IFJR42774 : 0.0.0.20120925_1011
C:\IBM\ProcessDesigner\v7.5 : JR44032_0.0.0.20120925_1001 : 7.5.1.1-WS-BPMPD-IFJR44032 : 0.0.0.20120925_1001
C:\IBM\ProcessDesigner\v7.5 : JR44357_0.0.0.20121126_1531 : 7.5.1.1-WS-BPMPD-IFJR44357 : 0.0.0.20121126_1531
C:\IBM\ProcessDesigner\v7.5 : JR45755_0.0.0.20121217_1010 : 7.5.1.1-WS-BPMPD-IFJR45755 : 0.0.0.20121217_1010

After

C:\IBM\ProcessDesigner\v7.5 : com.ibm.bpm.authoring.v75_7.5.1001.v20120914_0614 : IBM? Process Designer : 7.5.1.1
C:\IBM\ProcessDesigner\v7.5 : JR42774_0.0.0.20120925_1011 : 7.5.1.1-WS-BPMPD-IFJR42774 : 0.0.0.20120925_1011
C:\IBM\ProcessDesigner\v7.5 : JR44032_0.0.0.20120925_1001 : 7.5.1.1-WS-BPMPD-IFJR44032 : 0.0.0.20120925_1001
C:\IBM\ProcessDesigner\v7.5 : JR44357_0.0.0.20121126_1531 : 7.5.1.1-WS-BPMPD-IFJR44357 : 0.0.0.20121126_1531
C:\IBM\ProcessDesigner\v7.5 : JR45755_0.0.0.20121217_1010 : 7.5.1.1-WS-BPMPD-IFJR45755 : 0.0.0.20121217_1010
C:\IBM\ProcessDesigner\v7.5 : JR45939_0.0.0.20130508_1001 : 7.5.1.1-WS-BPMPD-IFJR45939 : 0.0.0.20130508_1001

IBM BPM - TeamWorksConfiguration.running.xml

This has caught me out several times in recent weeks.

TeamWorksConfiguration.running.xml is a fabulously useful file, as it provides one with the current state of a running IBM BPM environment, including all the configuration changes from 00Static.xml, 80Database.xml, 99Local.xml, 100Custom.xml etc.

For me, the file is found at the App Server profile level, in config/cells/nodes/servers: -

/opt/IBM/WebSphere/AppServer/profiles/AppSrv01/config/cells/ProcessCenter/nodes/AppSrv01.NodeProcessCenter.AppTarget.AppSrv01.Node.0/process-center/TeamWorksConfiguration.running.xml 

/opt/IBM/WebSphere/AppServer/profiles/Custom01/config/cells/localhostCell01/nodes/localhostNode01/servers/ProcessCenter.AppTarget.localhostNode01.0/process-center/TeamWorksConfiguration.running.xml

However, the key thing that I learned to my cost is ….

If one performs a Full Resynchronize of the WAS cell, the file gets removed.

This is because it's part of the node-level configuration, and a full resync. deletes all the Node configuration files and then replaces them from the Deployment Manager, from whence the cell is managed.

TeamWorksConfiguration.running.xml is absolutely, definitely a run-time file, which only ever exists at the Node level ( it's created by the AppTarget cluster which hosts the BPMN run-time and, if available, the PDW application/cluster )

Bottom line, if you ever sync. your nodes during normal operation, expect TeamWorksConfiguration.running.xml to disappear until the next time you restart your cluster(s)

WAS to DB2 - Null userid is not supported. ERRORCODE=-4461, SQLSTATE=42815

If you get: -

The test connection operation failed for data source Performance Data Warehouse ME data source on server nodeagent at node AppSrv01.Node with the following exception: java.sql.SQLException: [jcc][t4][10205][11234][3.61.65] Null userid is not supported. ERRORCODE=-4461, SQLSTATE=42815DSRA0010E: SQL State = 42815, Error Code = -4,461. View JVM logs for further details.

when testing JDBC data sources on a newly minted IBM BPM environment ( or, to be fair, IBM Business Monitor ), having just imported/generated the Deployment Environment, make sure that you've restarted your Deployment Manager and Node Agent(s), having first synchronised the node(s).

I keep hitting this problem, and keep on forgetting how darn simple the solution is.

At this stage, having imported the DE and sync'd the nodes, the WAS configuration still requires a restart to properly "take".

Easy :-)

Not RESTing on my laurels - IBM BPM Standard 7.5.1.1 and the ubiquitous localhost

I hit another localhost issue with Process Admin: -

https://rhel6.uk.ibm.com:9445/ProcessAdmin/ProcessAdmin/com.lombardisoftware.processadmin.ProcessAdmin/ProcessAdmin.jsp

when I clicked into the Process Inspector UI: -

which returned: -

Here's the detailed exception: -

com.google.gwt.http.client.RequestException: javax.servlet.ServletException: com.ibm.processinspector.rest.ProcessAdminRestException: Error communicating with server at com.ibm.processinspector.server.handlers.ProcessInstanceSearchHandler.handleRequest(ProcessInstanceSearchHandler.java:292) at com.ibm.processinspector.server.BPMInspectorRestFacade.doGet(BPMInspectorRestFacade.java:109) at javax.servlet.http.HttpServlet.service(HttpServlet.java:718) at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1658) at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1598) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:149) at com.lombardisoftware.servlet.SetUserTimeZoneFilter.doFilter(SetUserTimeZoneFilter.java:41) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125) at com.lombardisoftware.servlet.SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:35) at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:190) at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:125) at com.ibm.ws.webcontainer.filter.WebAppFilterChain._doFilter(WebAppFilterChain.java:80) at com.ibm.ws.webcontainer.filter.WebAppFilterManager.doFilter(WebAppFilterManager.java:908) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:935) at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:503) at com.ibm.ws.webcontainer.servlet.ServletWrapperImpl.handleRequest(ServletWrapperImpl.java:181) at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3954) at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:276) at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:942) at com.ibm.ws.webcontainer.WSWebContainer.handleRequest(WSWebContainer.java:1592) at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:186) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:453) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewRequest(HttpInboundLink.java:515) at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.processRequest(HttpInboundLink.java:306) at com.ibm.ws.http.channel.inbound.impl.HttpICLReadCallback.complete(HttpICLReadCallback.java:83) at com.ibm.ws.ssl.channel.impl.SSLReadServiceContext$SSLReadCompletedCallback.complete(SSLReadServiceContext.java:1784) at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165) at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217) at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161) at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:138) at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:204) at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:775) at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:905) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1646)Caused by: com.ibm.processinspector.rest.ProcessAdminRestException: Error communicating with server at com.ibm.processinspector.rest.ProcessAdminRestClient.doGetQuery(ProcessAdminRestClient.java:209) at com.ibm.processinspector.rest.ProcessAdminRestClient.doQuery(ProcessAdminRestClient.java:237) at com.ibm.processinspector.rest.ProcessInstanceSearchAPIImpl.getInstanceCount(ProcessInstanceSearchAPIImpl.java:169) at com.ibm.processinspector.rest.ProcessInstanceSearchAPIImpl.getInstanceStatusSummary(ProcessInstanceSearchAPIImpl.java:114) at com.ibm.processinspector.server.handlers.ProcessInstanceSearchHandler.createCounts(ProcessInstanceSearchHandler.java:73) at com.ibm.processinspector.server.handlers.ProcessInstanceSearchHandler.handleRequest(ProcessInstanceSearchHandler.java:281) ... 35 moreCaused by: javax.net.ssl.SSLException: hostname in certificate didn't match: <rhel6.uk.ibm.com> != <localhost> at org.apache.commons.httpclient.protocol.AbstractVerifier.verify(Unknown Source) at org.apache.commons.httpclient.protocol.BrowserCompatHostnameVerifier.verify(Unknown Source) at org.apache.commons.httpclient.protocol.AbstractVerifier.verify(Unknown Source) at org.apache.commons.httpclient.protocol.AbstractVerifier.verify(Unknown Source) at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(Unknown Source) at org.apache.commons.httpclient.protocol.SSLProtocolSocketFactory.createSocket(Unknown Source) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at com.ibm.processinspector.rest.ProcessAdminRestClient.doGetQuery(ProcessAdminRestClient.java:193) ... 40 moreError 500: javax.servlet.ServletException: javax.servlet.ServletException: com.ibm.processinspector.rest.ProcessAdminRestException: Error communicating with server

Again, here's the reason why: -

WAS Console ( ISC ) > Services > REST services > REST service providers

I changed the Host name or virtual host in a load-balanced environment from: -

localhost

to: -

rhel6.uk.ibm.com

Note - this needs to be done for BOTH REST endpoints: -

and restarted my AppTarget cluster.

Job done :-)

For the record, the five REST endpoints are summarised here: -

Process Designer to Process Center - More Fun, This Time with JMS QCF and TCF resources

Following on from my earlier post: -

Doh - Problem with Process Designer NOT connecting to Process Center

I realised that the issue that I was seeing with Process Designer > Process Center communication was also down to the JMS configuration within my WAS cell.

As mentioned, PD uses HTTP to get the endpoint, and then uses JMS for much of the rest of the work.

For this, we have both Queue connection factories (QCF) and Topic connection factories (TCF).

Both need to be configured to reflect the proper hostname of the endpoint i.e. that of the Process Center, and this needs to be resolvable from client (PD) to server (PC).

These are configured in the WAS console, via Resources > JMS > Queue Connection Factories / Topic Connection Factories.

There are five QCF resources: -

and a corresponding five TCF resources: -

In my case, all ten are wrong in one subtle way: -

I needed to change: -

localhost:7289:BootstrapSecureMessaging

to: -

rhel6.uk.ibm.com:7289:BootstrapSecureMessaging

where 7289 is the SIB_ENDPOINT_SECURE_ADDRESS for the Messaging Engine (JMS) cluster.

Doh - Problem with Process Designer NOT connecting to Process Center

More fun with IBM BPM Standard 7.5.1.1.

This time, Process Designer was failing to connect to my Process Center.

PD is on one machine ( Windows ) and PC is on another ( Linux ).

This is what I see: -

This happened regardless of whether I used HTTP: -

-Dcom.ibm.bpm.processcenter.url=http://rhel6.uk.ibm.com:9082

or HTTPS in eclipse.ini: -

-Dcom.ibm.bpm.processcenter.url=https://rhel6.uk.ibm.com:9445

Having faffed around for a while, I dug back through an internal IBM BPM forum ( hosted, of course, on IBM Connections ) and found: -

<snip>

I found the culprit.  The client had used "localhost" for the Linux host name.  Once I changed it to something reasonable, created a new profile for the PC, it worked first time.

</snip>

When I checked 99Local.xml : -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/localhostCell01/nodes/localhostNode01/servers/ProcessCenter.AppTarget.localhostNode01.0/process-center/config/system/99Local.xml

guess what I found ???

Yep, you guessed it: -

        <images-prefix>http://localhost:9082/teamworks</images-prefix>
        <portal-prefix>http://localhost:9082/portal</portal-prefix>                
        <repository-prefix>http://localhost:9082/ProcessCenter</repository-prefix>
        <servlet-prefix>http://localhost:9082/teamworks</servlet-prefix>
        <webapi-prefix>http://localhost:9082/webapi</webapi-prefix>
        <process-help-wiki-url-view>http://localhost:9082/processhelp/en/%TITLE%?teamworksTitle=%TEAMWORKS_TITLE%</process-help-wiki-url-view>
        <process-help-wiki-url-edit>http://localhost:9082/processhelp/en/Special:Edit?topic=%TITLE%&amp;teamworksTitle=%TEAMWORKS_TITLE%</process-help-wiki-url-edit>
        <portal-prefix>http://localhost:9082/portal</portal-prefix>
        <process-admin-prefix>http://localhost:9082/ProcessAdmin</process-admin-prefix>
        <teamworks-webapp-prefix>http://localhost:9082/teamworks</teamworks-webapp-prefix>
            <base-url>http://localhost:9082/teamworks/webservices</base-url>
            <default-namespace-uri>http://localhost:9082/schema/</default-namespace-uri>
        <coach-designer-xsl-url>http://localhost:9082/teamworks/coachdesigner/transform/CoachDesigner.xsl</coach-designer-xsl-url>
                <client-link>http://localhost:9082/teamworks</client-link>
        <repository-server-url>http://localhost:9082/ProcessCenter</repository-server-url>

Can you say "Doh!" ???

Now I need to override those settings in 100Custom.xml, resync my WAS cell, restart Process Center ( the AppTarget cluster, specifically ) and I should be good to go.

By the way, the reason that this is an issue ?

Process Designer uses HTTP/HTTPS to get to the bootstrap port of the AppTarget cluster member, gets the required info from BPM ( via 99Local / 100Custom ) and then connects via JMS to the Messaging Engine cluster.

Watch this space ….

IBM BPM Standard 7.5.1.1 - Fixing the Boostrap process by fixing DB2

I hit an interesting DB2-related issue whilst bootstrapping my IBM BPM databases: -

$ cd /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin

$ ./bootstrapProcessServerData.sh

which failed with: -

BUILD FAILED
/opt/IBM/WebSphere/AppServer/BPM/base/profile/actions/bootstrapData.ant:40: The following error occurred while executing this line:
/opt/IBM/WebSphere/AppServer/BPM/base/profile/actions/bootstrapData.ant:42: The following error occurred while executing this line:
/opt/IBM/WebSphere/AppServer/BPM/base/profile/actions/bootstrapData.ant:69: The following error occurred while executing this line:
/opt/IBM/WebSphere/AppServer/BPM/base/profile/actions/tw_init_bootstrap.ant:310: Java returned: 1

Total time: 54 seconds

Bootstrap failed

and, in the logs: -

$ cat /opt/IBM/WebSphere/AppServer/profiles/Dmgr01/logs/bootstrapProcesServerData.Tue-Aug-13-14.56.20-2013.log

INFO: CWLLG0378I: Importing snapshot 7.5.1.1 from process application Process Portal(TWP) :FAILED
13-Aug-2013 14:57:14 com.lombardisoftware.server.ejb.repositoryservices.RepositoryServicesCore importPackageInternal
INFO: CWLLG0162I: Import Failed
File /opt/IBM/WebSphere/AppServer/BPM/Lombardi/imports/process-portal.twx failed.
13-Aug-2013 14:57:14 com.lombardisoftware.utility.WLELoggerUtils logError
SEVERE: CWLLG2015E:  An unexpected runtime exception occurred. See exception for more details.  Error: PreparedStatementCallback; uncategorized SQLException for SQL [insert into LSW_RESOURCE_BUNDLE_KEY (RESOURCE_BUNDLE_KEY_ID,VERSION_ID,AKEY,VALUE,GUID,LAST_MODIFIED,RESOURCE_BUNDLE_ID,LAST_MODIFIED_BY_USER_ID) values (?,?,?,?,?,?,?,?)]; SQL state [57011]; error code [-964]; DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65; nested exception is com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65.
Throwable occurred: com.lombardisoftware.expimp.ExportImportException: PreparedStatementCallback; uncategorized SQLException for SQL [insert into LSW_RESOURCE_BUNDLE_KEY (RESOURCE_BUNDLE_KEY_ID,VERSION_ID,AKEY,VALUE,GUID,LAST_MODIFIED,RESOURCE_BUNDLE_ID,LAST_MODIFIED_BY_USER_ID) values (?,?,?,?,?,?,?,?)]; SQL state [57011]; error code [-964]; DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65; nested exception is com.ibm.db2.jcc.am.SqlException: DB2 SQL Error: SQLCODE=-964, SQLSTATE=57011, SQLERRMC=null, DRIVER=3.61.65

When I checked the corresponding DB2 log: -

$ cat /home/db2inst1/sqllib/db2dump/db2diag.log

I found: -

2013-08-13-14.57.07.540232+060 E948764E592         LEVEL: Error
PID     : 4484                 TID  : 140562648786688PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000          DB   : BPMDB
APPHDL  : 0-598                APPID: 127.0.0.1.34033.130813135710
AUTHID  : DB2INST1
EDUID   : 240                  EDUNAME: db2agent (BPMDB) 0
FUNCTION: DB2 UDB, data protection services, sqlpgResSpace, probe:2860
MESSAGE : ADM1823E  The active log is full and is held by application handle
          "598..".  Terminate this application by COMMIT, ROLLBACK or FORCE
          APPLICATION.

and: -

2013-08-13-14.57.07.540872+060 I949357E554         LEVEL: Error
PID     : 4484                 TID  : 140562648786688PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000          DB   : BPMDB
APPHDL  : 0-598                APPID: 127.0.0.1.34033.130813135710
AUTHID  : DB2INST1
EDUID   : 240                  EDUNAME: db2agent (BPMDB) 0
FUNCTION: DB2 UDB, data protection services, sqlpWriteLR, probe:6680
MESSAGE : ZRC=0x85100009=-2062548983=SQLP_NOSPACE
          "Log File has reached its saturation point"
          DIA8309C Log file was full.

This ties up with this IBM Technote: -

With IBM Business Process Manager products, IBM DB2 can display errors when under heavy process load

from which the solution is to increase the primary and secondary log file sizes.

This is what I did: -

$ db2 connect to bpmdb

$ db2 get snapshot for all on bpmdb > /tmp/snapshot.txt

which showed: -

Log space available to the database (Bytes)= 70955008
Log space used by the database (Bytes)     = 0
Secondary logs allocated currently         = 0

I updated the number of Secondary Logs as follows: -

$ db2 update db cfg for bpmdb using logsecond 12

and re-ran the bootstrap process.

This time it ran through to completion, which is nice :-)

Job done :-)