Thursday, 27 August 2015

WebSphere Application Server - High Availability Manager - What Is It Good For ?

I've just posted to the GWC blog here: -

This has come up in the past few weeks, as I've been configuring WebSphere Application Server (WAS) Network Deployment, aka WAS Full Profile, to store it's transaction log data in DB2 database tables.

This requirement exists because, for resilience and high availability, one typically deploys a WAS cell across multiple logical or physical nodes.

Want to know more ? You know where to go :-)

Building and deploying business monitor models for IBM Business Process Manager V8.5 processes

I saw this on Twitter and thought of .... well, everyone :-)

Learn how you can generate a monitor model from IBM® Business Process Manager V8.5 and deploy the monitor model in an IBM Business Monitor server installed in a separate WebSphere® cell. This content is part of the IBM Business Process Management Journal.

IBM Integration Designer (hereafter called Integration Designer) provides the capability to generate a monitor model from your IBM Business Process Manager (BPM) project. This tutorial will show you how to generate such a monitor model and subsequently deploy the monitor model to an IBM Business Monitor server, which runs in a cell separate from the IBM BPM hosting cell. The process model developer will need to turn the tracking on in the Process Designer for elements that will need to be monitored in the process.

Tuesday, 25 August 2015

Podcasts app in iOS - Sunk, not syncing

OK, so I managed to break, and then fix, the Podcasts app in iOS 8.4.1 on an iPhone 5s.

The lesson learned ? Don't do this in iTunes 

as it seems to stop the Podcasts app from downloading updates to the podcast feed.

In fact, it's worse than that ( it's dead, Jim ) in that the app hangs altogether, and then completely crashes.

I tried the IT Crowd trick: -

but to no avail.

The solution appeared to be related to the fact that BOTH iTunes and Podcasts were trying to sync.

Once I told iTunes to stop ( and synced the iPhone, and rebooted the phone again !! ), the Podcasts app just started working, and now tomorrow's commute is sorted.

Thursday, 20 August 2015

Comment lines: Encrypting WebSphere Application Server system passwords — if you insist

This from my IBM colleague, Martin Lansche: -

IBM® WebSphere® Application Server stores system passwords in files that are simply encoded. To support clients who want to implement their own password storage mechanism, WebSphere Application Server has provided a System Programming Interface (SPI) to do so. Encrypting these system passwords can provide some marginal additional security benefits for a specific class of vulnerability (poorly implemented application code), but it cannot provide defense against a malicious programmer or a determined attack. If you insist on implementing a password encryption solution, here is one example of how you could achieve this using the SPI. This content is part of the IBM WebSphere Developer Technical Journal.

Wednesday, 19 August 2015

"Invalid operation: result set is closed" with IBM BPM 8.5.5

This is in the context of an IBM Business Process Manager Advanced installation on AIX.

I'd noticed a spurious exception in the First Failure Data Capture (FFDC) logs of both members of the AppCluster: -

19/08/15 09:44:01:200 BST]     FFDC Exception:org.springframework.jdbc.UncategorizedSQLException ProbeId:934 org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [select reference_id, dep_path_id, target_type, target_id, target_summary_data from lsw_po_reference where branch_id = ? and start_seq_num <= ? and end_seq_num > ?]; SQL state [null]; error code [-4470]; [jcc][t4][10120][10898][4.19.26] Invalid operation: result set is closed. ERRORCODE=-4470, SQLSTATE=null; nested exception is [jcc][t4][10120][10898][4.19.26] Invalid operation:
result set is closed. ERRORCODE=-4470, SQLSTATE=null

which, whilst not necessarily representing a major issue was somewhat of an annoyance.

I dug around online, and found this thread on IBM developerWorks Answers ( dW Answers ): -

This, in part, suggested that the problem might be mitigated by changing a Custom Property of the JDBC data source used by the AppCluster, specifically jdbc/TeamWorksDB, as described here: -

In essence, it directs one to set resultSetHoldability to 1 rather than the default of not set.

I did this, and ended up with a cluster that refused to start, due: -

[19/08/15 08:34:36:288 BST] FFDC ProbeId:865 [jcc][t4][10434][12579][4.11.69] Invalid operation: ResultSet holdability HOLD_CURSORS_OVER_COMMIT is not allowed on an XA connection. ERRORCODE=-4476, SQLSTATE=null

The dW Answers post also suggested that this iFix: -

would mitigate the problem.

Alas, I already had this applied to my BPM environment, as it was released ~12 months ago in August 2014.

Finally, it suggested that the DB2 JDBC driver might needed to be updated. I checked, and the BPM box was running an older version of the driver. I copied updated drivers ( db2jcc.jar and db2jcc4.jar ) across from the DB2 server, but to no avail. The problem persisted ....

For the record, the latest DB2 JDBC drivers can be found here: -

 I actually had the latest drivers applied to DB2, as they're consistent between and -

I then found this dW Answers post: -

which says, in part: -

For DB2, these driver properties can impact the behavior, although they don't really change things for the BPM XA transactions. 
resultSetHoldability Specifies whether cursors remain open after a commit operation. The data type of this property is int. Valid values are:

Leave cursors open after a commit operation.
This value is not valid for a connection that is part of a distributed (XA) transaction.

Close cursors after a commit operation.

downgradeHoldCursorsUnderXa Specifies whether cursors that are defined WITH HOLD can be opened under XA connections.

If downgradeHoldCursorsUnderXa is set to false, then an error is thrown for this scenario. 
If downgradeHoldCursorsUnderXa is set to true, a cursor that is defined WITH HOLD can be opened under an XA connection. However, the cursor has the following restrictions:

• When the cursor is opened under an XA connection, the cursor does not have WITH HOLD behavior. The cursor is closed at XA End.

• A cursor that is open before XA Start on a local transaction is closed at XA Start.

Therefore, I set resultSetHoldability back to 1 ( HOLD_CURSORS_OVER_COMMIT ) and also added a new Custom Property, downgradeHoldCursorsUnderXa, which 
I set to true.

This last most recent change appears to have fixed / mitigated the issue.

In summary, it was a combination of the JR50863 iFix AND the two JDBC data source Custom Properties. The JDBC driver version appears to make no difference, and I did roll back to the shipped version, at least for now.

Nice :-)

Monday, 17 August 2015

RestCEIAuthLocator: security.xml not found

Hmmm, so I saw this during the startup of a previously more-than-happy WAS cluster member, specifically a member of my IBM BPM  8.5.5 Support cluster: -

[17/08/15 08:33:46:944 BST] 00000001 ModelMgr      I   WSVR0801I: Initializing all server configuration models
[17/08/15 08:33:47:042 BST] 00000001 Ffdc          I logIncident FFDC1003I: FFDC Incident emitted on /opt/ibm/WebSphereProfiles/PSCell1AppSrv01/logs/ffdc/ffdc.2593104506022466010.txt 50
[17/08/15 08:33:47:046 BST] 00000001 SystemOut     O RestCEIAuthLocator: security.xml not found: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException
[17/08/15 08:33:47:046 BST] 00000001 ModelMgr      I   WSVR0801I: Initializing all server configuration models
[17/08/15 08:33:47:196 BST] 00000001 ModelMgr      I   WSVR0801I: Initializing all server configuration models
[17/08/15 08:33:47:306 BST] 00000001 wtp           E initializeExtensions ERROR: Bindings & Extensions failed to initialize.
[17/08/15 08:33:47:374 BST] 00000001 ModelMgr      I   WSVR0800I: Initializing core configuration models
[17/08/15 08:33:47:379 BST] 00000001 WsServerImpl  E   WSVR0009E: Error occurred during startup

This was a surprise, given that the cluster member had been starting OK previously, BUT I had been making some configuration changes last week, when I enabled transaction/partner logging in DB2.

I did the usual things, including turning it off and on again (!), and also comparing / contrasting the configuration between it and it's happily working counterpart. I also fully resynchronised the node configuration from the Deployment Manager.

But to no avail .....

Even Google did NOT have the answer this time - I did find this: -

but, alas, the answer wasn't the one for which I was looking: -

Hi, It looks like your App Server may be corrupted.  

[5/19/15 12:01:28:260 GMT+02:00] 00000001 SystemOut     O RestCEIAuthLocator: security.xml not found: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.IllegalStateException: java.lang.reflect.InvocationTargetException

the security.xml file is missing for some reason.  Try doing a full sync from the node to the dmgr.  If that does not work, you can probably copy the file over from the dmgr to the node.

as I'd already checked that (a) security.xml was definitely available to the failing JVM and (b) that it was identical to the cell-level copy on the Deployment Manager AND the other (working) node.

I then checked a few internal IBM sites, and found something that caught my eye, pertaining to a BPM JVM failing to start AFTER IBM Tivoli Composite Application Manager (ITCAM) had been installed.

Given that we DO use ITCAM's successor, IBM Application Performance Management, I read on further, and found reference to this ITCAM APAR: -

Whilst this references a specific ITCAM fix, it also mentioned a circumvention: -

Local Fix/Workaround:
   Clearing the OSGi cache using the
   prior to restarting the server instance resolves the issue.

I took the chance, and ran: -


which cleared the OSGI cache for ALL the JVMs in that particular profile, including the failing SupClusterMember1, and then tried to start the JVM again.

This time .... yep, it came up clean and green.

So I don't know precisely what happened, and I will be checking the ITCAM fix with my APM colleagues, in case there's a corresponding APM patch.

But I have a working cluster, and that's all good then.

Wednesday, 12 August 2015

IBM Business Process Manager operation overview, Part 1: Topology, security, basic administration, and monitoring

Newly published on IBM developerWorks: -

The maintenance of a clustered server environment for IBM® Business Process Manager (BPM) that interacts with many back-end and front-end systems and services can be a challenging task. Administrators can follow the IBM Business Process Manager operation overview series to guide their daily operation work. Part 1 helps administrators with topology, security, and basic administrative and monitoring operations.