HTTP Load Balancing - F5 Big IP and IBM WebSphere Application Server

One of my colleagues drew my attention to these: -

Deploying F5 with IBM WebSphere 7 [PDF]

Deploying the BIG-IP LTM with IBM WebSphere 8 [PDF]

As we're rolling out IBM BPM 8.5.5 ( based upon WAS 8.5 ) with Big IP shortly, this is going to be extremely useful.

Python vs. Jython - The case of the mysterious argument ...

I'm writing a Jython script to manage IBM BPM security roles, as I mentioned previously: -

Things that make you go Hmmmmm #2772 - Updating RunAs Roles in IBM BPM Advanced 8.5.5

when I discovered that Jython and Python treat command-line arguments differently.

So here's two scripts: -

foo.py

#!/usr/bin/python

import sys

a = sys.argv[0]
b = sys.argv[1]
c = sys.argv[2]

print a + " " + b + " " + c

bar.py

#!/usr/bin/python

import sys

a = sys.argv[1]
b = sys.argv[2]
c = sys.argv[3]

print a + " " + b + " " + c

and here's what happens when I run each of them in Python: -

python foo.py Hello World !

foo.py Hello World

python bar.py Hello World !

Hello World !

And now here's what happens when I run each of them in Jython: -

/opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -conntype none -f foo.py Hello World !

WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[Hello, World, !]"

Hello World !

/opt/IBM/WebSphere/AppServer/bin/wsadmin.sh -conntype none -f bar.py Hello World !

WASX7357I: By request, this scripting client is not connected to any server process. Certain configuration and application operations will be available in local mode.
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[Hello, World, !]"
WASX7017E: Exception received while running file "bar.py"; exception information: com.ibm.bsf.BSFException: exception from Jython:
Traceback (innermost last):
  File "<string>", line 7, in ?
IndexError: index out of range: 3

In other words, Python starts counting the arguments from one ( which seems quite logical ), where argument zero is the name of the script.

By contrast, Jython starts counting from zero, hence the exception when I run bar.py i.e. the script looks for an argument indexed 3 whereas it only received arguments 0, 1 and 2 - hence the index out of range error.

Interesting, eh ?

IBM Business Process Manager operation overview, Part 2: Maintenance and migration

This just in: -

Part 2 of this series gives an overview of the maintenance and migration steps that administrators for IBM® Business Process Manager (IBM BPM) complete in their daily operation work. The maintenance of a clustered server environment for IBM BPM that interacts with many back-end and front-end systems and services can be a challenging task. The overview and tips help administrators plan for and successfully maintain and migrate IBM BPM systems. This content is part of the IBM Business Process Management Journal.

IBM Business Process Manager operation overview, Part 2: Maintenance and migration

This follows on from: -

IBM Business Process Manager operation overview, Part 1: Topology, security, basic administration, and monitoring

The maintenance of a clustered server environment for IBM® Business Process Manager (BPM) that interacts with many back-end and front-end systems and services can be a challenging task. Administrators can follow the IBM Business Process Manager operation overview series to guide their daily operation work. Part 1 helps administrators with topology, security, and basic administrative and monitoring operations.

Control VMware Fusion from the Command Line

From a rather useful blog post: -

Control VMware Fusion from the Command Line

I was looking for a way to script the suspension of a couple of VMware Fusion VMs.

This is with what I ended up: -

Suspend the IBM BPM VM

/Applications/VMware\ Fusion.app/Contents/Library/vmrun -T ws suspend ~/Documents/Virtual\ Machines.localized/BPM_LDAP.vmwarevm/BPM_LDAP.vmx

Suspend the MS Active Directory VM

/Applications/VMware\ Fusion.app/Contents/Library/vmrun -T ws suspend ~/Documents/Virtual\ Machines.localized/W2K8.vmwarevm/W2K8.vmx

This is in the context of an earlier Mac OS X sleep/wake issue about which I had previously blogged: -

VMware Fusion on Mac OS X - Sleep, no wake-ee

Getting up to speed with Docker and IBM Bluemix

This is most definitely a Work-In-Progress: -

Containerized IBM Business Process Manager Advanced 8.5.6 Process Center

Doing it the Docker way: Standalone web server

WebSphere full profile Installed in a Docker container

Dockerfile that creates a docker image with WebSphere for Developers installed and IBM JDK 7

Get Started with Docker for Mac OS X

Github - cloudfoundry/cli

IBM Bluemix - Setting up the IBM Containers CLI

Pulling repository registry.ng.bluemix.net gives Authentication Failure

so please watch this space.

Things that make you go Hmmmmm #2772 - Updating RunAs Roles in IBM BPM Advanced 8.5.5

I'm in the process of documenting the migration of an IBM BPM Advanced 8.5.5 environment from the out-of-the-box file-based user registry to an external registry, MS Active Directory.

I'm going to write the entire process up more fully, once I've proven it to actually work.

However, one of the required steps took me a wee while to fully comprehend, and I obviously misunderstood the documentation.

This is the Knowledge Center page to which I refer: -

Defining RunAs roles user assignments for system applications

which says in part: -

...

IBM Business Process Manager provides the following applications that contain users that are mapped to the RunAs roles:

• IBM_BPM_PerformanceDW_supportDeploymentTarget

• IBM_BPM_Teamworks_applicationDeploymentTarget

• BPEContainer__applicationDeploymentTarget

• TaskContainer__applicationDeploymentTarget

...

...

Note: The bpmModifyMapRunAsRole script is used to update the IBM_BPM_Teamworks*.ear and IBM_BPM_PerformanceDW*.ear applications. It does not update other IBM Business Process Manager applications.

To define the RunAs roles user assignments, run the bpmModifyMapRunAsRole script, which is located at install_root/util/Security/bpmModifyMapRunAsRole.py.

...

So I read and absorbed this, and wrote the following series of commands: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -username wasadmin -password passw0rd -f /opt/IBM/WebSphere/AppServer/util/Security/bpmModifyMapRunAsRole.py -usr DepEnvAdmin -pwd P455w0rd -clusterName SupCluster -applicationName IBM_BPM_PerformanceDW_SupCluster

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd -f /opt/IBM/WebSphere/AppServer/util/Security/bpmModifyMapRunAsRole.py -usr DepEnvAdmin -pwd P455w0rd -clusterName AppCluster -applicationName IBM_BPM_Teamworks_AppCluster

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd -f /opt/IBM/WebSphere/AppServer/util/Security/bpmModifyMapRunAsRole.py -usr DepEnvAdmin -pwd P455w0rd -clusterName AppCluster -applicationName BPEContainer_AppCluster

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd -f /opt/IBM/WebSphere/AppServer/util/Security/bpmModifyMapRunAsRole.py -usr DepEnvAdmin -pwd P455w0rd -clusterName AppCluster -applicationName TaskContainer_AppCluster

each of which ran (!) and returned: -

*************************************************************************************
Start: Mapping run as roles to users

Mapping users for cluster profile.

End: Mapping run as roles to users
*************************************************************************************

However .....

The change didn't "take" and I could see no differences to the RunAs user mappings for each of the four applications.

Which wasn't nice.

So I read the actual Jython script: -

/opt/IBM/WebSphere/AppServer/util/Security/bpmModifyMapRunAsRole.py 

and saw, in part: -

...

if(applicationName == ''):
    appName = "IBM_BPM_Teamworks_" + targetName
    options = '[-MapRunAsRolesToUsers [[twem ' + userName + ' ' + password + '] [twuser ' + userName + ' ' + password + ']]]'
    AdminApp.edit(appName, options)

    appName = "IBM_BPM_PerformanceDW_" + targetName
    options = '[-MapRunAsRolesToUsers [[tw-loader ' + userName + ' ' + password + ']]]'
    AdminApp.edit(appName, options)

else:
     if(applicationName == 'Teamworks'):
      appName = "IBM_BPM_Teamworks_" + targetName
      options = '[-MapRunAsRolesToUsers [[twem ' + userName + ' ' + password + '] [twuser ' + userName + ' ' + password + ']]]'
      AdminApp.edit(appName, options)
     elif(applicationName == 'PerformanceDW'):
      appName = "IBM_BPM_PerformanceDW_" + targetName
      options = '[-MapRunAsRolesToUsers [[tw-loader ' + userName + ' ' + password + ']]]'
      AdminApp.edit(appName, options)

...

...

AdminConfig.save();
...

Interestingly, I never actually saw the output from AdminConfig.save() either, which rang alarum bells.

In other words, the script was looking for JUST the application name, NOT the fully qualified application name ( with the cluster name appended ) e.g. PerformanceDW rather than IBM_BPM_PerformanceDW_SupCluster.

Once I amended my commands: -

I started to see more meaningful messages: -

*************************************************************************************
Start: Mapping run as roles to users

Mapping users for cluster profile.

ADMA5075I: Editing of application IBM_BPM_PerformanceDW_SupCluster started.
ADMA5058I: Application and module versions are validated with versions of deployment targets.
ADMA5005I: The application IBM_BPM_PerformanceDW_SupCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application IBM_BPM_PerformanceDW_SupCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application IBM_BPM_PerformanceDW_SupCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application IBM_BPM_PerformanceDW_SupCluster is configured in the WebSphere Application Server repository.
ADMA5113I: Activation plan created successfully.
ADMA5011I: The cleanup of the temp directory for application IBM_BPM_PerformanceDW_SupCluster is complete.
ADMA5076I: Application IBM_BPM_PerformanceDW_SupCluster edited successfully. The application or its web modules may require a restart when a save is performed.
End: Mapping run as roles to users
*************************************************************************************

*************************************************************************************
Start: Mapping run as roles to users

Mapping users for cluster profile.

ADMA5075I: Editing of application IBM_BPM_Teamworks_AppCluster started.
ADMA5058I: Application and module versions are validated with versions of deployment targets.
ADMA5005I: The application IBM_BPM_Teamworks_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application IBM_BPM_Teamworks_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application IBM_BPM_Teamworks_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application IBM_BPM_Teamworks_AppCluster is configured in the WebSphere Application Server repository.
ADMA5113I: Activation plan created successfully.
ADMA5011I: The cleanup of the temp directory for application IBM_BPM_Teamworks_AppCluster is complete.
ADMA5076I: Application IBM_BPM_Teamworks_AppCluster edited successfully. The application or its web modules may require a restart when a save is performed.
End: Mapping run as roles to users
*************************************************************************************

etc.

*AND* more importantly, the change was actually effective: -

which IS nice.

However, the script does NOT touch the BPEContainer and TaskContainer applications, as can be seen from the IF/ELSE logic above; it only works on the PerformanceDW and TeamWorks applications :-(

I ended up writing my own, modified, version of this script, specifically for the two extra BPM applications: -

daveModifyMapRunAsRole.py 
userName = sys.argv[0]
password = sys.argv[1]
clusterName = sys.argv[2]

targetName = clusterName

appName = "BPEContainer_" + targetName
options = '[-MapRunAsRolesToUsers [[JMSAPIUser ' + userName + ' ' + password + '] [AdminJobUser ' + userName + ' ' + password + ']]]'
AdminApp.edit(appName, options)

appName = "TaskContainer_" + targetName
options = '[-MapRunAsRolesToUsers [[AdminJobUser ' + userName + ' ' + password + '] [EscalationUser ' + userName + ' ' + password + ']]]'
AdminApp.edit(appName, options)

AdminConfig.save()
AdminNodeManagement.syncActiveNodes()

which I ran as follows: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/bin/wsadmin.sh -lang jython -user wasadmin -password passw0rd -f daveModifyMapRunAsRole.py DepEnvAdmin P455w0rd AppCluster

with the following results: -

WASX7209I: Connected to process "dmgr" on node Dmgr using SOAP connector;  The type of process is: DeploymentManager
WASX7303I: The following options are passed to the scripting environment and are available as arguments that are stored in the argv variable: "[DepEnvAdmin, P455w0rd, AppCluster]"
ADMA5075I: Editing of application BPEContainer_AppCluster started.
ADMA5058I: Application and module versions are validated with versions of deployment targets.
ADMA5005I: The application BPEContainer_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application BPEContainer_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application BPEContainer_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application BPEContainer_AppCluster is configured in the WebSphere Application Server repository.
ADMA5113I: Activation plan created successfully.
ADMA5011I: The cleanup of the temp directory for application BPEContainer_AppCluster is complete.
ADMA5076I: Application BPEContainer_AppCluster edited successfully. The application or its web modules may require a restart when a save is performed.
ADMA5075I: Editing of application TaskContainer_AppCluster started.
ADMA5058I: Application and module versions are validated with versions of deployment targets.
ADMA5005I: The application TaskContainer_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application TaskContainer_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application TaskContainer_AppCluster is configured in the WebSphere Application Server repository.
ADMA5005I: The application TaskContainer_AppCluster is configured in the WebSphere Application Server repository.
ADMA5113I: Activation plan created successfully.
ADMA5011I: The cleanup of the temp directory for application TaskContainer_AppCluster is complete.
ADMA5076I: Application TaskContainer_AppCluster edited successfully. The application or its web modules may require a restart when a save is performed.
---------------------------------------------------------------
 AdminNodeManagement:        Synchronize the active nodes
 Usage: AdminNodeManagement.syncActiveNodes()
 Return: If the command is successfully invoked, a value of 1 is returned. 
---------------------------------------------------------------
 
Node1

I'm going to be writing this up more fully once I've run through the end-to-end process a few more times ....

IBM BPM Advanced - Problems with the Internal Document Store - Be careful what you change, and when

Ooops, I broke it, it was me :-)

Whilst migrating a BPM Advanced 8.5.5 environment from a file-based user registry ( fileRegistry.xml ) to Microsoft Active Directory ( via LDAP ), I inadvertently broke the IBM BPM Document Store ( based upon FileNet ).

I started seeing exceptions such as: -

[24/09/15 16:16:41:190 BST] 0000006d EmbeddedECMIn E   CWTDS1100E: An error occurred while validating or creating the default configuration for the IBM BPM document store.
                                 com.ibm.bpm.embeddedecm.exception.UserMissesWritePermissionException: CWTDS0022E: The configuration was changed in a way that the technical user 'DepEnvAdmin' of the IBM BPM document store fails to change the object 'Domain'.
Explanation: The technical user defined in the BPM role type 'EmbeddedECMTechnicalUser' is not permitted to perform changes on an object.
Action: Revert the recent configuration changes. Ensure that the user defined by the BPM role type 'EmbeddedECMTechnicalUser' has access to the object. Verify this using the admin task 'getDocumentStoreStatus'.

in the AppCluster SystemOut.log.

When I tried to work with the Document Store Authorisation service: -

AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -add uid=DepEnvAdmin,o=defaultWIMFileBasedRealm]')

I saw: -

WASX7015E: Exception running command: "AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -add uid=DepEnvAdmin,o=defaultWIMFileBasedRealm]')"; exception information:
com.ibm.bpm.embeddedecm.exception.UnexpectedFailureException: com.ibm.bpm.embeddedecm.exception.UnexpectedFailureException: CWTDS0000E: An unexpected failure occurred. Details: 'FNRCE0057: E_READ_ONLY: The method failed because an object or property is read-only.'
Explanation: An exception was thrown.
Action: Check the server log files.

The root cause ?

I'd switched from the file-based user - deadline - to the LDAP user - DepEnvAdmin ( that's the subject of another blog post ) *BUT* I neglected to "tell" the Document Store before I deleted the deadmin user from the file-based registry: -

AdminTask.deleteUser('[-uniqueName uid=deadmin,o=defaultWIMFileBasedRealm]')

I dug around, and found this IBM Technote: -

IBM Business Process Manager (BPM) cannot communicate with the IBM BPM document store

and this: -

Administering the technical user for the IBM BPM document store

Thankfully, I *DID* have a backup of the file-based registry: -

/opt/IBM/WebSphere/AppServer/profiles/Dmgr01/config/cells/PCCell1/fileRegistry.xml

<?xml version="1.0" encoding="UTF-8"?>
<xml.type:datagraph xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:wim="http://www.ibm.com/websphere/wim"
    xmlns:xml.type="commonj.sdo">
  <changeSummary xmlns=""/>
  <wim:Root>
    <wim:entities xsi:type="wim:PersonAccount">
      <wim:identifier externalId="be7f5abc-c40e-41ff-81e6-f7d1cfbaf214" externalName="uid=wasadmin,o=defaultWIMFileBasedRealm"
          uniqueId="be7f5abc-c40e-41ff-81e6-f7d1cfbaf214" uniqueName="uid=wasadmin,o=defaultWIMFileBasedRealm"/>
      <wim:parent>
        <wim:identifier uniqueName="o=defaultWIMFileBasedRealm"/>
      </wim:parent>
      <wim:createTimestamp>2015-09-22T10:27:30.890Z</wim:createTimestamp>
      <wim:password>U0hBLTE6dXhtbjllY2o3YjQzOm9lY0tsbXh2ZXVGeWZXY00xTUl5NlNNR0x5UT0K</wim:password>
      <wim:uid>wasadmin</wim:uid>
      <wim:cn>wasadmin</wim:cn>
      <wim:sn>wasadmin</wim:sn>
    </wim:entities>
    <wim:entities xsi:type="wim:PersonAccount">
      <wim:identifier externalId="ffe99ec4-a1c6-4d52-93f9-02d7d80e55b6" externalName="uid=deAdmin,o=defaultWIMFileBasedRealm"
          uniqueId="ffe99ec4-a1c6-4d52-93f9-02d7d80e55b6" uniqueName="uid=deAdmin,o=defaultWIMFileBasedRealm"/>
      <wim:parent>
        <wim:identifier uniqueName="o=defaultWIMFileBasedRealm"/>
      </wim:parent>
      <wim:createTimestamp>2015-09-22T11:41:43.017+01:00</wim:createTimestamp>
      <wim:password>U0hBLTE6aGxsdjlsZWtlMW1qOlhVQ2tNUm9rUmR6bnM5WWRVTTEvUkNNVkFFND0K</wim:password>
      <wim:uid>deAdmin</wim:uid>
      <wim:cn>deAdmin</wim:cn>
      <wim:sn>deAdmin</wim:sn>
    </wim:entities>
  </wim:Root>
</xml.type:datagraph>

I restored this from my backup, and restarted the AppCluster AND the Deployment Manager.

This time around the cluster started without exception, and I was able to access the Document Store using WSadmin: -

AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -list]')

"Authorization on the domain for the IBM BPM document store\nAuthorization on the object store for the IBM BPM document store\nCWTDS2035I: Access is granted to the IBM BPM document store object store 'uid=deAdmin,o=defaultWIMFileBasedRealm' with access mask '838,205,440'."

AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -add CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com]')

"CWTDS1005I: Domain access for the IBM BPM document store granted for principal 'CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com'.\nCWTDS1006I: Object store access for the IBM BPM document store granted for principal 'CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com'.\nCWTDS2027I: The access to the IBM BPM document store was successfully modified."

AdminTask.maintainDocumentStoreAuthorization('[-deName De1 -list]')

"Authorization on the domain for the IBM BPM document store\nCWTDS2034I: Access is granted to the IBM BPM document store domain 'CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com' with access mask '459,267'.\nCWTDS2034I: Access is granted to the IBM BPM document store domain 'uid=deAdmin,o=defaultWIMFileBasedRealm' with access mask '459,267'.\nAuthorization on the object store for the IBM BPM document store\nCWTDS2035I: Access is granted to the IBM BPM document store object store 'CN=DepEnvAdmin,CN=Users,DC=uk,DC=ibm,DC=com' with access mask '838,205,440'.\nCWTDS2035I: Access is granted to the IBM BPM document store object store 'uid=deAdmin,o=defaultWIMFileBasedRealm' with access mask '838,205,440'."

I don't yet have a precise sequence of events BUT it's important to get this Document Store (ECM) stuff sorted BEFORE making any other substantive changes to WAS/BPM security i.e. updating J2C Authentication Aliases, deleting users etc.

Definitely read this: -

Administering the technical user for the IBM BPM document store

first :-)

Building IBM Business Monitor models with the freedom of the Dynamic Event Framework

This from IBM developerWorks via Twitter: -

Building IBM Business Monitor models with the freedom of the Dynamic Event Framework

Define both events and payload with a flexible framework

The Dynamic Event Framework is a new event transport mechanism in IBM® Business Process Manager (BPM) and IBM Business Monitor starting in version 8.5.5. This tutorial uses a simple Claims processing scenario to provide step-by-step instructions for creating and detailing a monitor model to leverage Dynamic Event Framework events.

Koralling Kerberos Keytabs

So, on my Mac using OS X 10.10, I can happily use the ktutil command to view a Kerberos key tab: -

ktutil -k bpm855.keytab list

bpm855.keytab:

Vno  Type              Principal                  Aliases
  3  arcfour-hmac-md5  HTTP/bpm855.uk.ibm.com@UK  

but, on Red Hat Linux, this doesn't seem to work quite as well - I need to use several commands: -

ktutil 
ktutil:  rkt bpm855.keytab
ktutil:  list
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    3                HTTP/bpm855.uk.ibm.com@UK
ktutil:  quit

Thankfully, the internet came to the rescue: -

How to automate ktutil to immediately list keytab entries?

Create a shell macro: -

rkt() { echo -e "read_kt $1\nlist\nquit" | ktutil | grep -v "^ktutil:";}

Read the key tab: -

rkt bpm855.keytab 

ktutil:  read_kt bpm855.keytab
slot KVNO Principal
---- ---- ---------------------------------------------------------------------
   1    3                HTTP/bpm855.uk.ibm.com@UK

which is nice.

Thanks Internet, I owe you :-)

WebSphere Liberty Profile - An Installation In Many Parts

See what's available to install

/opt/IBM/InstallationManager/eclipse/tools/imcl listAvailablePackages -repositories /home/wasadmin/Repository/LibertyCore/,/home/wasadmin/Repository/Java7/

com.ibm.websphere.liberty.v85_8.5.5000.20130514_1313

com.ibm.websphere.IBMJAVA.v70_7.0.4001.20130510_2103

com.ibm.websphere.liberty.IBMJAVA.v70_7.0.4001.20130510_2103

/opt/IBM/InstallationManager/eclipse/tools/imcl listAvailablePackages -repositories /home/wasadmin/Repository/LibertyCore/,/home/wasadmin/Repository/LibertyFP,/home/wasadmin/Repository/Java7/

com.ibm.websphere.liberty.v85_8.5.5000.20130514_1313

com.ibm.websphere.liberty.BASE.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.BASETRIAL.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.DEVELOPERS.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.DEVELOPERSILAN.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.EXPRESS.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.ND.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.NDTRIAL.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.TRIAL.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.WEBENAB.v85_8.5.5006.20150610_1804

com.ibm.websphere.liberty.v85_8.5.5006.20150610_1804

com.ibm.websphere.IBMJAVA.v70_7.0.4001.20130510_2103

com.ibm.websphere.liberty.IBMJAVA.v70_7.0.4001.20130510_2103

Perform a dummy installation

/opt/IBM/InstallationManager/eclipse/tools/imcl install com.ibm.websphere.liberty.v85 com.ibm.websphere.liberty.IBMJAVA.v70 -repositories /home/wasadmin/Repository/LibertyCore/,/home/wasadmin/Repository/LibertyFP,/home/wasadmin/Repository/Java7/ -installationDirectory /opt/IBM/WebSphere/Liberty -sharedResourcesDirectory /opt/IBM/IMShared -acceptLicense -skipInstall /tmp/snafu

Installed com.ibm.websphere.liberty.v85_8.5.5006.20150610_1804 to the /opt/IBM/WebSphere/Liberty directory.

Installed com.ibm.websphere.liberty.IBMJAVA.v70_7.0.4001.20130510_2103 to the /opt/IBM/WebSphere/Liberty directory.

Perform the real installation

/opt/IBM/InstallationManager/eclipse/tools/imcl install com.ibm.websphere.liberty.v85 com.ibm.websphere.liberty.IBMJAVA.v70 -repositories /home/wasadmin/Repository/LibertyCore/,/home/wasadmin/Repository/LibertyFP,/home/wasadmin/Repository/Java7/ -installationDirectory /opt/IBM/WebSphere/Liberty -sharedResourcesDirectory /opt/IBM/IMShared -acceptLicense

Installed com.ibm.websphere.liberty.v85_8.5.5006.20150610_1804 to the /opt/IBM/WebSphere/Liberty directory.

Installed com.ibm.websphere.liberty.IBMJAVA.v70_7.0.4001.20130510_2103 to the /opt/IBM/WebSphere/Liberty directory.

Validate installation

/opt/IBM/InstallationManager/eclipse/tools/imcl listInstalledPackages

com.ibm.cic.agent_1.8.2000.20150303_1526

com.ibm.websphere.liberty.IBMJAVA.v70_7.0.4001.20130510_2103

com.ibm.websphere.liberty.v85_8.5.5006.20150610_1804

/opt/IBM/WebSphere/Liberty/bin/productInfo version

Product name: WebSphere Application Server

Product version: 8.5.5.6

Product edition: LIBERTY_CORE

/opt/IBM/WebSphere/Liberty/bin/server version

WebSphere Application Server 8.5.5.6 (1.0.9.cl50620150610-1749) on IBM J9 VM, version pxa6470sr4fp1ifx-20130423_02 (SR4 FP1+IV38579+IV38399+IV40208) (en_GB)

/opt/IBM/WebSphere/Liberty/java/java_1.7_64/bin/java -version

java version "1.7.0"

Java(TM) SE Runtime Environment (build pxa6470sr4fp1ifix-20130423_02(SR4 FP1+IV38579+IV38399+IV40208))

IBM J9 VM (build 2.6, JRE 1.7.0 Linux amd64-64 Compressed References 20130421_145945 (JIT enabled, AOT enabled)

J9VM - R26_Java726_SR4_FP1_2_20130421_2353_B145945

JIT  - r11.b03_20130131_32403ifx4

GC   - R26_Java726_SR4_FP1_2_20130421_2353_B145945_CMPRSS

J9CL - 20130422_145945)

JCL - 20130225_01 based on Oracle 7u13-b08

WebSphere Liberty Profile - Patching

We have choices, we have lots of choices: -

WebSphere Liberty Profile and IBM Installation Manager - A Marriage

From this: -

Installing and uninstalling Liberty using downloaded files and archives

To try out the Liberty profile and use the Liberty profile to develop applications that run on the WebSphere® Application Server Liberty profile or full profile, you can download a no-charge, unsupported edition from the WASdev download page.

To use the Liberty profile in a production environment with guaranteed service levels and IBM® support, you must purchase WebSphere Application Server (base), WebSphere Application Server Network Deployment, WebSphere Application Server Express®, or WebSphere Application Server Liberty Core. The Liberty profile is included with these editions and can also be downloaded separately, as an edition-specific Java archive (JAR) file, from Passport Advantage® Online. The associated service is available from Fix Central. If you download and install the Liberty profile from an unsupported JAR or ZIP file, you can later purchase a supported edition and upgrade the license for your existing installation.

I'll be shortly penning an article on the installation of Liberty and Java using IBM Installation Manager .......

Aide Memoire - IBM BPM - Moving from File-Based Registry to LDAP

I'm writing this down as I'll need in the not-too-distant future.

On my current project, we have BPM Advanced 8.5.5 ( plus Business Monitor 8.5.5 and ODM Rules 8.7.0 ), all of which are using the out-of-the-box File-Based Registry for a limited number of end users ( mainly admins, developers and testers ).

We are, of course, migrating this to a real user registry ( Microsoft Active Directory ) in the near future.

To that end, I will need to remember that, for BPM, there are some specific things that need to be done, including: -

Administering the technical user for the IBM BPM document store

When working with the IBM BPM document store, there are multiple scenarios that require a technical user (system user). The technical user is an identity that the system can use to act on its own. For example, a run-as technical user is required for creating default configurations for the domain, object store, and document class definition. A technical user is also required when IBM Business Process Manager connects to the IBM BPM document store using Content Management Interoperability Service (CMIS).

updateBPMAliasesAndRunAsRolesPasswords command

As one of my colleagues succinctly put it: -

In case you can create the exact same users (that is: same user, same password) in LDAP, then you should be able to just add LDAP and remove fileRegistry. It is important to understand the from an internal document store's perspective, a user with the same name is "just a user with the same name". It is NOT the same user and thus does not have access. Make sure to follow the instruction in http://www-01.ibm.com/support/knowledgecenter/SSFTN5_8.5.6/com.ibm.wbpm.admin.doc/topics/tbpmdocauth.html > "Reconfiguring the user registry" to avoid locking yourself out of the embedded document store.

and: -

In case the LDAP accounts' passwords (and/or usernames) are different from those in fileRegistry you also need to update authentication aliases and EJB run-as roles. I see documentation for an AdminTask for that purpose

WAS and DB2 HADR - What I'm reading

So we have this: -

Configuring messaging engine and server behavior when a data store connection is lost

in the context of the sib.msgstore.jdbcFailoverOnDBConnectionLoss custom property. This defaults to true in recent versions of WAS, meaning: -

The high availability manager stops the messaging engine and its hosting application server when the next core group service Is alive check takes place (the default value is 120 seconds). If a node agent is monitoring the server, and you have enabled automatic restart in the monitoring policy for the server, the server restarts. The messaging engine starts when an appropriate server is available.

whereas I've set it to false meaning: -

The messaging engine continues to run and accept work, and periodically attempts to regain the connection to the data store. If work continues to be submitted to the messaging engine while the data store is unavailable, the results can be unpredictable, and the messaging engine can be in an inconsistent state when the data store connection is restored.

plus this: -

CWSIS1538I, CWSIS1546I: Messaging engine failover operation fails due to lost database locks

and this: -

TCP/IP keepalive settings and related DB2 registry variables

in the context of this: -

/etc/sysctl.conf 

...

# Added to support WAS > DB2 keepalives etc.

net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_retries2 = 2
...

Plus these: -

Problem diagnosis with the IBM Data Server Driver for JDBC and SQLJ

Common IBM Data Server Driver for JDBC and SQLJ properties for all supported database products

Operation of automatic client reroute for connections to DB2 for Linux, UNIX, and Windows from Java clients

in the context of the JDBC client parameter db2.jcc.outputDirectory which controls the location/creation of a cache file - jccServerListCache.bin - which is used by JDBC client applications ( in my case, Cognos Business Intelligence ) to know where to go when the primary ( and configured ) DB2 server goes down.

I'll be blogging more about the latter shortly ....

WebSphere Application Server - SIbus Resilience with DB2 HADR - There and back again

Yet more on DB2 HADR: -

DB2 HADR - Updating Host Names

DB2 HADR - A Clash of Primaries

DB2 HADR - More intel. about what's goin' on

WebSphere Application Server 8.5.5 and DB2 10.5 - High Availability for Disaster Recovery - Like Manuel, I learn ....

DB2 - HADR Performance Tuning and Diagnostics

I'm testing DB2 HADR using a pair of VMs, running v10.5.0.5, and a 3rd VM running WAS 8.5.5.2 ( this is Business Monitor 8.5.5 but that's not important right now ).

I'm trying to get a handle (!)  on Messaging Engine ( SIbus ) failover between primary and standby DB2 boxes.

I've configured the ME Custom Property sib.msgstore.jdbcFailoverOnDBConnectionLoss = false  rather than the WAS 8.5 default of true.

I've also tuned the TCP/IP keep-alives on all three VMs to: -

net.ipv4.tcp_keepalive_time = 60
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_retries2 = 2

and yet SIB DB failover does not ... failover.

I start the MECluster ( which hosts the SIB ME ), and it happily connects to the primary DB ( db2one.uk.ibm.com ).

When I do the DB2 HADR takeover on db2two.uk.ibm.com, I see: -

[18/09/15 21:18:42:274 BST] 0000006a ConnectionEve A   J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adapter for resource jdbc/wbm/MonitorDatabase. The exception is: com.ibm.db2.jcc.am.ClientRerouteException: [jcc][t4][2027][11212][4.11.69] A connection failed but has been re-established. The host name or IP address is "db2one" and the service name or port number is 60,006.

in SystemOut.log, before it connects: -

[18/09/15 21:19:02:232 BST] 00000069 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1537I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=7E0AFF46E21AA864, has acquired an exclusive lock on the data store.

However, when i reverse the takeover ( back to d2one.uk.ibm.com ), I see: -

[18/09/15 21:19:22:269 BST] 000000a6 WSJccConnecti W   DSRA8650W: Error closing a JDBC child wrapper, com.ibm.ws.rsadapter.jdbc.WSJccPreparedStatement@c693be3a
com.ibm.db2.jcc.am.SqlException: [jcc][10120][10943][4.11.69] Invalid operation: statement is closed. ERRORCODE=-4470, SQLSTATE=null

...

[18/09/15 21:19:22:278 BST] 000000a6 ConnectionEve W   J2CA0206W: A connection error occurred.  To help determine the problem, enable the Diagnose Connection Usage option on the Connection Factory or Data Source. This is the multithreaded access detection option. Alternatively check that the Database or MessageProvider is available.
[18/09/15 21:19:22:279 BST] 000000a6 ConnectionEve A   J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adapter for resource jdbc/wbm/MonitorDatabase. The exception is: com.ibm.db2.jcc.am.DisconnectNonTransientConnectionException: [jcc][t4][2030][11211][4.11.69] A communication error occurred during operations on the connection's underlying socket, socket input stream,
or socket output stream.  Error location: Reply.fill() - insufficient data (-1).  Message: Insufficient data. ERRORCODE=-4499, SQLSTATE=08001
[18/09/15 21:19:42:234 BST] 00000069 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1594I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=7E0AFF46E21AA864, has lost the lock on the data store.
[18/09/15 21:19:42:240 BST] 00000069 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1538I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=7E0AFF46E21AA864, is attempting to obtain an exclusive lock on the data store.

...

In other words, it fails from the configured primary db2one to the configured standby db2two but not back again.

This is what I have for the primary DB: -

and this is what I have for the standby: -

I then made some progress :-)

So this is what I have on db2two: -

db2 list db directory

 System Database Directory

 Number of entries in the directory = 2

Database 1 entry:

 Database alias                       = COGNOS
 Database name                        = COGNOS
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              = IBM Cognos Content Store
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2one
 Alternate server port number         = 60006

Database 2 entry:

 Database alias                       = MONITOR
 Database name                        = MONITOR
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2one
 Alternate server port number         = 60006

and this is what I have on db2one: -

db2 list db directory

 System Database Directory

 Number of entries in the directory = 2

Database 1 entry:

 Database alias                       = COGNOS
 Database name                        = COGNOS
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              = IBM Cognos Content Store
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            = db2two
 Alternate server port number         = 60006

Database 2 entry:

 Database alias                       = MONITOR
 Database name                        = MONITOR
 Local database directory             = /home/db2inst1
 Database release level               = 10.00
 Comment                              =
 Directory entry type                 = Indirect
 Catalog database partition number    = 0
 Alternate server hostname            =
 Alternate server port number         =

Spot the difference ?

Yep, the DB2 catalog on db2one has no Alternate server settings for the MONITOR DB, and guess with what DB I'm having problems ?

Added to that, I see this: -

2015-09-19-07.28.52.105153+060 I61711243E775         LEVEL: Warning
PID     : 2100                 TID : 140240484296448 PROC : db2sysc 0
INSTANCE: db2inst1             NODE : 000            DB   : MONITOR
APPHDL  : 0-13494              APPID: 192.168.33.100.56862.150919073103
AUTHID  : DB2USER1             HOSTNAME: db2one.uk.ibm.com
EDUID   : 156                  EDUNAME: db2agent (MONITOR) 0
FUNCTION: DB2 UDB, High Availability Disaster Recovery, hdrCheckDb, probe:18200
MESSAGE : SQL1776N  The command cannot be issued on an HADR database. Reason
          code = "1".
DATA #1 : Hex integer, 4 bytes
0x00000000
DATA #2 : sqeApplication_acbInfo, PD_TYPE_sqeApplication_acbInfo, 4 bytes
x0
DATA #3 : String, 50 bytes
Connections are not allowed on a standby database.

in ~/sqllib/db2dump/db2diag.log.

I updated the Alternate server settings on db2one as follows: -

db2 "update alternate server for database monitor using hostname db2two port 60006"

and restarted DB2: -

db2stop force

db2start

and ... guess what ?

Yep, takeover now works both ways: -

From db2two to db2one

[19/09/15 07:34:11:758 BST] 000000bf ConnectionEve W   J2CA0206W: A connection error occurred.  To help determine the problem, enable the Diagnose Connection Usage option on the Connection Factory or Data Source. This is the multithreaded access detection option. Alternatively check that the Database or MessageProvider is available.

[19/09/15 07:34:11:759 BST] 000000bf ConnectionEve A   J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adapter for resource jdbc/wbm/MonitorDatabase. The exception is: com.ibm.db2.jcc.am.ClientRerouteException: [jcc][t4][2027][11212][4.11.69] A connection failed but has been re-established. The host name or IP address is "db2two.uk.ibm.com" and the service name or port number is 60,006.

Special registers may or may not be re-attempted (Reason code = 1). ERRORCODE=-4498, SQLSTATE=08506
[19/09/15 07:34:31:487 BST] 00000068 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1594I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=A35AD93CE23D2EA7, has lost the lock on the data store.
[19/09/15 07:34:31:488 BST] 00000068 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1538I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=A35AD93CE23D2EA7, is attempting to obtain an exclusive lock on the data store.
[19/09/15 07:34:31:655 BST] 00000154 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1545I: A single previous owner was found in the messaging engine's data store, ME_UUID=FE28F02EE5B5F496, INC_UUID=A35AD93CE23D2EA7
[19/09/15 07:34:31:659 BST] 00000068 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1537I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=A35AD93CE23D2EA7, has acquired an exclusive lock on the data store.

From db2one to db2two

[19/09/15 07:37:31:801 BST] 00000154 ConnectionEve W   J2CA0206W: A connection error occurred.  To help determine the problem, enable the Diagnose Connection Usage option on the Connection Factory or Data Source. This is the multithreaded access detection option. Alternatively check that the Database or MessageProvider is available.

[19/09/15 07:37:31:802 BST] 00000154 ConnectionEve A   J2CA0056I: The Connection Manager received a fatal connection error from the Resource Adapter for resource jdbc/wbm/MonitorDatabase. The exception is: com.ibm.db2.jcc.am.ClientRerouteException: [jcc][t4][2027][11212][4.11.69] A connection failed but has been re-established. The host name or IP address is "db2one.uk.ibm.com" and the service name or port number is 60,006.
Special registers may or may not be re-attempted (Reason code = 1). ERRORCODE=-4498, SQLSTATE=08506
[19/09/15 07:37:51:666 BST] 00000068 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1594I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=A35AD93CE23D2EA7, has lost the lock on the data store.
[19/09/15 07:37:51:668 BST] 00000068 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1538I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=A35AD93CE23D2EA7, is attempting to obtain an exclusive lock on the data store.
[19/09/15 07:37:51:787 BST] 0000015a SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1545I: A single previous owner was found in the messaging engine's data store, ME_UUID=FE28F02EE5B5F496, INC_UUID=A35AD93CE23D2EA7
[19/09/15 07:37:51:797 BST] 00000068 SibMessage    I   [MONITOR.BAMCell1.Bus:MECluster.000-MONITOR.BAMCell1.Bus] CWSIS1537I: The messaging engine, ME_UUID=FE28F02EE5B5F496, INC_UUID=A35AD93CE23D2EA7, has acquired an exclusive lock on the data store.

Webcast replay: WebSphere Application Server - Service Integration Bus Messaging Engine Data Store Connectivity Problems and Solutions

This is worth a read and re-read: -

This WebSphere Support Technical Exchange is designed to describe how Service Integration Bus Messaging Engines connect to data stores and how to recover from unexpected disconnections from data stores.

Webcast replay: WebSphere Application Server - Service Integration Bus Messaging Engine Data Store Connectivity Problems and Solutions

It's especially relevant to me right now as I work out how to get a WAS 8.5.5 Messaging Engine to happily accept the failover of its SIB database tables from one DB2 HADR box to another.

I'm currently buried in TCP/IP retry parameters ... which is nice.

Sings "I feel a blog post arisin' !"

Top 5 editor's picks for IBM Business Monitor

I saw this on Twitter: -

Top 5 editor's picks for IBM Business Monitor

Here are the top 5 editor's picks of IBM® Business Monitor content that is published on developerWorks. I selected content that helps you to get Business Monitor up and running on a recent version, to use dashboards, and to understand options for integrating with IBM Business Process Manager.

Top 5 editor's picks for IBM Business Monitor

DB2 HADR - More intel. about what's goin' on

So I've previously written about HADR: -

DB2 HADR - Updating Host Names

WebSphere Application Server 8.5.5 and DB2 10.5 - High Availability for Disaster Recovery - Like Manuel, I learn ....

DB2 - HADR Performance Tuning and Diagnostics

so here's today's news from the metaphorical coal face: -

The command: -

db2pd -alldbs

gives one a HUGE amount of information about DB2 e.g.

...

Database Member 0 -- Database MONITOR -- Active -- Up 0 days 00:08:40 -- Date 2015-09-16-13.15.27.843535

Work Action Sets:
Address            ActionSetID ActionSetName                  ClassSetID  ObjectType ObjectID   
0x00007F4922917F20 2147483647  SYSDEFAULTUSERWAS              2147483647  b               3          

Work Actions:
Address            ActionSetID ActionID    ClassID     Type       ActualsOptions RefObjectID 
0x00007F4922DE9D40 2147483647  2147483647  2147483647  M               None       4           

Database Member 0 -- Database MONITOR -- Active -- Up 0 days 00:08:40 -- Date 2015-09-16-13.15.27.843574

Work Class Sets:
Address            ClassSetID  ReferenceCounter
0x00007F4922A17620 2147483647  1          

Work Classes:

Address              = 0x00007F4922D17C80 
ClassSetId           = 2147483647 
ClassId              = 2147483647 
ClassName            = SYSMANAGEDQUERIES             
Work Class Attributes: 
  Work Type          = 2                   
  Timeron Cost: 
    From Value       = 150000              
    To Value         = 0                   

Database Member 0 -- Database MONITOR -- Active -- Up 0 days 00:08:40 -- Date 2015-09-16-13.15.27.844343

...

This can be narrowed down to JUST HADR: -

db2pd -alldbs -hadr

...

Database Member 0 -- Database COGNOS -- Active -- Up 0 days 02:25:44 -- Date 2015-09-16-13.16.48.990144

                            HADR_ROLE = PRIMARY
                          REPLAY_TYPE = PHYSICAL
                        HADR_SYNCMODE = SYNC
                           STANDBY_ID = 1
                        LOG_STREAM_ID = 0
                           HADR_STATE = PEER
                           HADR_FLAGS = 
                  PRIMARY_MEMBER_HOST = db2two
                     PRIMARY_INSTANCE = db2inst1
                       PRIMARY_MEMBER = 0
                  STANDBY_MEMBER_HOST = db2one
                     STANDBY_INSTANCE = db2inst1
                       STANDBY_MEMBER = 0
                  HADR_CONNECT_STATUS = CONNECTED
             HADR_CONNECT_STATUS_TIME = 16/09/2015 10:51:06.138265 (1442397066)
          HEARTBEAT_INTERVAL(seconds) = 15
                     HEARTBEAT_MISSED = 0
                   HEARTBEAT_EXPECTED = 582
                HADR_TIMEOUT(seconds) = 60

...

or even more narrowly: -

db2pd -hadr -db monitor

Database Member 0 -- Database MONITOR -- Active -- Up 0 days 00:10:37 -- Date 2015-09-16-13.17.24.757338

                            HADR_ROLE = PRIMARY
                          REPLAY_TYPE = PHYSICAL
                        HADR_SYNCMODE = SYNC
                           STANDBY_ID = 1
                        LOG_STREAM_ID = 0
                           HADR_STATE = PEER
                           HADR_FLAGS = 
                  PRIMARY_MEMBER_HOST = db2two
                     PRIMARY_INSTANCE = db2inst1
                       PRIMARY_MEMBER = 0
                  STANDBY_MEMBER_HOST = db2one
                     STANDBY_INSTANCE = db2inst1
                       STANDBY_MEMBER = 0
                  HADR_CONNECT_STATUS = CONNECTED
             HADR_CONNECT_STATUS_TIME = 16/09/2015 13:06:49.283326 (1442405209)
          HEARTBEAT_INTERVAL(seconds) = 15
                     HEARTBEAT_MISSED = 0
                   HEARTBEAT_EXPECTED = 42
                HADR_TIMEOUT(seconds) = 60
        TIME_SINCE_LAST_RECV(seconds) = 5
             PEER_WAIT_LIMIT(seconds) = 0
           LOG_HADR_WAIT_CUR(seconds) = 0.000
    LOG_HADR_WAIT_RECENT_AVG(seconds) = 0.000000
   LOG_HADR_WAIT_ACCUMULATED(seconds) = 0.000
                  LOG_HADR_WAIT_COUNT = 0
SOCK_SEND_BUF_REQUESTED,ACTUAL(bytes) = 0, 19800
SOCK_RECV_BUF_REQUESTED,ACTUAL(bytes) = 0, 87380
            PRIMARY_LOG_FILE,PAGE,POS = S0000159.LOG, 0, 737788609
            STANDBY_LOG_FILE,PAGE,POS = S0000159.LOG, 0, 737788609
                  HADR_LOG_GAP(bytes) = 0
     STANDBY_REPLAY_LOG_FILE,PAGE,POS = S0000159.LOG, 0, 737788609
       STANDBY_RECV_REPLAY_GAP(bytes) = 0
                     PRIMARY_LOG_TIME = 16/09/2015 13:02:48.000000 (1442404968)
                     STANDBY_LOG_TIME = 16/09/2015 13:02:48.000000 (1442404968)
              STANDBY_REPLAY_LOG_TIME = 16/09/2015 13:02:48.000000 (1442404968)
         STANDBY_RECV_BUF_SIZE(pages) = 4298
             STANDBY_RECV_BUF_PERCENT = 0
           STANDBY_SPOOL_LIMIT(pages) = 25600
                STANDBY_SPOOL_PERCENT = 0
                   STANDBY_ERROR_TIME = NULL
                 PEER_WINDOW(seconds) = 120
                      PEER_WINDOW_END = 16/09/2015 13:19:19.000000 (1442405959)
             READS_ON_STANDBY_ENABLED = N

However, we've also got this: -

db2 get snapshot for database manager

            Database Manager Snapshot

Node type                                      = Enterprise Server Edition with local and remote clients
Instance name                                  = db2inst1
Number of members in DB2 instance              = 1
Database manager status                        = Active

Product name                                   = DB2 v10.5.0.5
Service level                                  = s141128 (IP23633)

Private Sort heap allocated                    = 0
Private Sort heap high water mark              = 0
Post threshold sorts                           = Not Collected
Piped sorts requested                          = 6
Piped sorts accepted                           = 6

Start Database Manager timestamp               = 16/09/2015 10:43:14.922175
Last reset timestamp                           =
Snapshot timestamp                             = 16/09/2015 13:17:57.551177

Remote connections to db manager               = 0
Remote connections executing in db manager     = 0
Local connections                              = 0
Local connections executing in db manager      = 0
Active local databases                         = 2

High water mark for agents registered          = 14
Agents registered                              = 5
Idle agents                                    = 0

Committed private Memory (Bytes)               = 29753344

Switch list for member 0
Buffer Pool Activity Information  (BUFFERPOOL) = OFF
Lock Information                        (LOCK) = OFF
Sorting Information                     (SORT) = OFF
SQL Statement Information          (STATEMENT) = OFF
Table Activity Information             (TABLE) = OFF
Take Timestamp Information         (TIMESTAMP) = ON  16/09/2015 10:43:14.922175
Unit of Work Information                 (UOW) = OFF

Agents assigned from pool                      = 354
Agents created from empty pool                 = 33
Agents stolen from another application         = 0
High water mark for coordinating agents        = 11
Hash joins after heap threshold exceeded       = 0
OLAP functions after heap threshold exceeded   = 0

Total number of gateway connections            = 0
Current number of gateway connections          = 0
Gateway connections waiting for host reply     = 0
Gateway connections waiting for client request = 0
Gateway connection pool agents stolen          = 0

Node FCM information corresponds to            = 0
Free FCM buffers                               = 128
Total FCM buffers                              = 128
Free FCM buffers low water mark                = 128
Maximum number of FCM buffers                  = 8192
Free FCM channels                              = 128
Total FCM channels                             = 128
Free FCM channels low water mark               = 127
Maximum number of FCM channels                 = 8192


Memory usage for database manager:

  Node number                                  = 0
    Memory Pool Type                           = Other Memory
       Current size (bytes)                    = 64815104
       High water mark (bytes)                 = 66781184
       Configured size (bytes)                 = 106627072

  Node number                                  = 0
    Memory Pool Type                           = FCMBP Heap
       Current size (bytes)                    = 851968
       High water mark (bytes)                 = 851968
       Configured size (bytes)                 = 851968

  Node number                                  = 0
    Memory Pool Type                           = Database Monitor Heap
       Current size (bytes)                    = 262144
       High water mark (bytes)                 = 524288
       Configured size (bytes)                 = 393216

and, for the server that's CURRENTLY hosting both PRIMARY databases: -

...

Active utilities for database manager:

  Database name                                = MONITOR

     Node number                               = 0
     Type                                      = ROLLFORWARD RECOVERY
     ID                                        = 13
     Description                               = Database Rollforward Recovery
     Start time                                = 16/09/2015 13:05:55.638776

  Database name                                = COGNOS

     Node number                               = 0
     Type                                      = ROLLFORWARD RECOVERY
     ID                                        = 3
     Description                               = Database Rollforward Recovery
     Start time                                = 16/09/2015 10:03:37.498669
...

which is nice.

Added to this, we do have: -

db2 list applications

which shows what is connecting to what: -

Auth Id  Application    Appl.      Application Id                                                 DB       # of
         Name           Handle                                                                    Name    Agents
-------- -------------- ---------- -------------------------------------------------------------- -------- -----
DB2USER1 db2jcc_applica 348        192.168.33.100.40667.150916122323                              MONITOR  1    
DB2USER1 db2jcc_applica 359        192.168.33.100.40670.150916122334                              MONITOR  1    
DB2USER1 db2jcc_applica 358        192.168.33.100.40669.150916122333                              MONITOR  1    
DB2USER1 db2jcc_applica 357        192.168.33.100.40668.150916122332                              MONITOR  1    

Auth Id  Application    Appl.      Application Id                                                 DB       # of
         Name           Handle                                                                    Name    Agents
-------- -------------- ---------- -------------------------------------------------------------- -------- -----
DB2USER1 db2jcc_applica 381        192.168.33.100.53243.150916123702                              COGNOS   1    
DB2USER1 IBM_C10_2_1_CM 374        192.168.33.100.53193.150916123559                              COGNOS   1    
DB2USER1 db2jcc_applica 380        192.168.33.100.53242.150916123701                              COGNOS   1    
DB2USER1 IBM_C10_2_1_CM 373        192.168.33.100.53192.150916123558                              COGNOS   1    
DB2USER1 IBM_C10_2_1_CM 379        192.168.33.100.53198.150916123606                              COGNOS   1    
DB2USER1 IBM_C10_2_1_CM 372        192.168.33.100.53178.150916123553                              COGNOS   1    
DB2USER1 IBM_C10_2_1_CM 378        192.168.33.100.53197.150916123603                              COGNOS   1    
DB2USER1 db2jcc_applica 384        192.168.33.100.53246.150916123706                              COGNOS   1    
DB2USER1 IBM_C10_2_1_CM 377        192.168.33.100.53196.150916123602                              COGNOS   1    
DB2USER1 IBM_C10_2_1_CM 376        192.168.33.100.53195.150916123601                              COGNOS   1    

etc.